Tasks / Strategy

Competitor analysis

Can the model find where a competitor can actually hurt you, from win/loss and churn data, rather than list features?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 63% were usable with at most a quick edit.

Reliably right

  1. Respects explicit constraints100% pass
    Delivers a competitive analysis memo to Dan within 900 words, addressing all requested topics without proposing a full feature-by-feature match.
    Opus 5.5 · Claude · The competitor isn't who you think
  2. Reads where Crestline is heading100% pass
    Uses Crestline's hiring (payroll, multi-entity, accountant partnerships) to infer direction, flags the partner-programme risk, and adjusts the response accordingly.
    GPT-6 Astra · ChatGPT · Free forever, funded
  3. Produces the required deliverable97% pass
    The memo is complete, in the right form for the CEO, within the word limit, and actionable with light edits.
    Opus 5.5 · Claude · The competitor isn't who you think

Where it slips

  1. Uses the supplied evidence correctly56% pass
    Two factual claims are wrong: the number of unconnected multi-user accounts going to Crestline is 40, not 42, and the number of connected 6-20 cancellations is 5, not 4.
    Sonnet 5.5 · API · Free forever, funded
  2. Gets the base of every number right61% pass
    Derived figures for unconnected multi-user cancellations (42 vs actual 40) and connected 6-20 cancellations (4 vs actual 5) are incorrect.
    Sonnet 5.5 · API · Free forever, funded
  3. Separates what a rival can copy66% pass
    Does not explicitly separate advantages a well-funded competitor could copy from those it couldn't, nor anchor the response in durable advantages with evidence.
    GPT-6 Luna · API · The competitor isn't who you think

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Competes with what customers really choose

    Does the analysis treat the status quo (spreadsheets, manual work, doing nothing) as a competitor, and draw at least one implication from where competitors are investing rather than only from today's features?

    Passes when Analyses the status quo, strengths included, and gives a direction-of-travel insight that affects the recommendation.

  2. Separates what a rival can copy

    Does the analysis separate advantages a well-funded competitor could copy (features, price, being first) from ones it couldn't easily (switching costs, networks, relationships), and build the response on the latter?

    Passes when Names which advantages are copyable and which aren't, with evidence, and anchors the response in the durable ones.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the VP Product at Harbourline. Crestline raised $120M in August and launched a free plan for solo businesses. Our board chair wants to know whether we should launch a free plan for everyone to match it (the note is below). Write the board a competitive analysis of no more than 1,200 words: where Crestline can hurt us and where it can't, with the evidence; which of our advantages it could copy and which it couldn't; where it's heading; and what we should do. The files are attached.

What the model was given6 items: About Harbourline, board_chair_note.md, customer_base.md, cancellations_aug_sep.csv (every account that cancelled since Crestline's launch), cancellation_summary.md, crestline_public_info.md
About HarbourlineBookkeeping, invoicing and expenses for small businesses, with a partner network that connects customers to their accounting firm.
board_chair_note.md1 lines · Download
From the board chair: "Crestline's free tier is an existential threat. Should we launch a free plan for everyone to match it? We need a view on where they can hurt us and where they can't."
customer_base.md9 lines · Download
# Harbourline customers (end of September)

| Account size | Paying accounts | Share connected to an accounting firm | Average revenue per month | Annual churn before Crestline's free tier |
|---|---|---|---|---|
| 1 user | 11,400 | 9% | $14 | 31% |
| 2-5 users | 4,900 | 46% | $52 | 14% |
| 6-20 users | 1,300 | 81% | $170 | 6% |
…
cancellations_aug_sep.csv (every account that cancelled since Crestline's launch)cancelled_account,account_users,connected_to_accounting_firm,mrr_usd,went_to H-1200,1,no,14,Crestline (free tier) H-1201,1,no,10,Crestline (free tier) H-1202,2-5,no,58,Crestline (free tier) H-1203,1,no,11,Crestline (free tier) H-1204,1,no,17,Crestline (free tier) H-1205,1,no,15,Spreadsheets / no tool H-1206,1,no,15,Spreadsheets / no tool H-1207,1,no,9,Crestline (free tier) H-1208,2-5,no,66,Crestline (free tier) H-1209,2-5,no,44,Crestline (free tier) H-1210,1,no,11,Crestline (free tier) H-1211,1,no,9,Crestline (free tier) H-1212,1,no,12,Crestline (free tier) H-1213,1,no,16,Crestline (free tier) H-1214,2-5,yes,73,Crestline (free tier) H-1215,1,no,9,Crestline (free tier) H-1216,2-5,no,56,Crestline (free tier) H-1217,1,no,14,Crestline (free tier) H-1218,2-5,no,47,Business closed H-1219,1,no,16,Crestline (free tier) H-1220,6-20,no,177,Crestline (free tier) H-1221,2-5,yes,51,Crestline (free tier) H-1222,2-5,no,54,Business closed H-1223,6-20,no,168,Crestline (free tier) H-1224,1,no,18,Crestline (free tier) H-1225,1,no,13,Spreadsheets / no tool H-1226,1,no,11,Crestline (free tier) H-1227,1,no,15,Crestline (free tier) H-1228,1,no,12,Crestline (free tier) H-1229,1,no,13,Crestline (free tier) H-1230,2-5,no,51,Business closed H-1231,1,no,18,Crestline (free tier) H-1232,1,no,16,Spreadsheets / no tool H-1233,1,no,12,Crestline (free tier) H-1234,1,no,16,Spreadsheets / no tool H-1235,1,no,12,Spreadsheets / no tool H-1236,1,no,13,Crestline (free tier) H-1237,1,no,17,Spreadsheets / no tool H-1238,6-20,yes,188,Business closed H-1239,1,no,17,Crestline (free tier) H-1240,1,no,15,Crestline (free tier) H-1241,6-20,yes,238,Crestline (free tier) H-1242,1,no,11,Spreadsheets / no tool H-1243,2-5,no,66,Crestline (free tier) H-1244,2-5,no,63,Business closed H-1245,6-20,no,199,Crestline (free tier) H-1246,1,no,18,Crestline (free tier) H-1247,1,no,11,Crestline (free tier) H-1248,1,no,15,Crestline (free tier) H-1249,2-5,no,60,Crestline (free tier) H-1250,2-5,no,66,Crestline (free tier) H-1251,2-5,no,34,Crestline (free tier) H-1252,1,no,12,Crestline (free tier) H-1253,1,no,18,Crestline (free tier) H-1254,1,no,18,Spreadsheets / no tool H-1255,2-5,no,31,Crestline (free tier) H-1256,1,no,17,Crestline (free tier) H-1257,1,no,16,Crestline (free tier) H-1258,1,no,18,Spreadsheets / no tool H-1259,1,no,9,Crestline (free tier) H-1260,1,no,15,Crestline (free tier) H-1261,1,no,18,Crestline (free tier) H-1262,6-20,yes,170,Business closed H-1263,1,no,18,Crestline (free tier) H-1264,2-5,no,41,Business closed H-1265,1,no,14,Spreadsheets / no tool H-1266,1,no,14,Crestline (free tier) H-1267,1,no,12,Crestline (free tier) H-1268,6-20,no,112,Crestline (free tier) H-1269,1,no,9,Crestline (free tier) H-1270,2-5,no,71,Business closed H-1271,1,no,12,Crestline (free tier) H-1272,1,no,11,Crestline (free tier) H-1273,1,no,15,Crestline (free tier) H-1274,1,no,9,Crestline (free tier) H-1275,6-20,no,209,Crestline (free tier) H-1276,2-5,no,36,Crestline (free tier) H-1277,1,no,13,Crestline (free tier) H-1278,1,no,11,Crestline (free tier) H-1279,1,no,15,Crestline (free tier) H-1280,1,no,18,Spreadsheets / no tool H-1281,1,no,16,Crestline (free tier) H-1282,1,no,17,Crestline (free tier) H-1283,1,no,9,Crestline (free tier) H-1284,1,no,10,Crestline (free tier) H-1285,6-20,yes,128,Business closed H-1286,1,no,17,Crestline (free tier) H-1287,1,no,12,Spreadsheets / no tool H-1288,2-5,no,39,Crestline (free tier) H-1289,1,no,14,Crestline (free tier) H-1290,1,no,12,Crestline (free tier) H-1291,1,no,16,Crestline (free tier) H-1292,2-5,no,62,Crestline (free tier) H-1293,1,no,12,Crestline (free tier) H-1294,1,no,12,Crestline (free tier) H-1295,1,no,9,Spreadsheets / no tool H-1296,1,no,11,Crestline (free tier) H-1297,1,no,16,Crestline (free tier) H-1298,1,no,17,Crestline (free tier) H-1299,1,no,13,Spreadsheets / no tool H-1300,1,no,13,Crestline (free tier) H-1301,1,no,17,Crestline (free tier) H-1302,1,no,12,Crestline (free tier) H-1303,1,no,17,Crestline (free tier) H-1304,1,no,14,Crestline (free tier) H-1305,1,no,9,Crestline (free tier) H-1306,1,no,15,Crestline (free tier) H-1307,1,no,16,Crestline (free tier) H-1308,1,no,16,Crestline (free tier) H-1309,1,no,12,Crestline (free tier) H-1310,2-5,no,47,Business closed H-1311,2-5,no,78,Crestline (free tier) H-1312,1,no,11,Crestline (free tier) H-1313,1,no,14,Crestline (free tier) H-1314,1,no,15,Spreadsheets / no tool H-1315,1,no,14,Spreadsheets / no tool H-1316,1,no,9,Crestline (free tier) H-1317,2-5,no,52,Business closed H-1318,1,no,13,Crestline (free tier) H-1319,2-5,no,43,Business closed H-1320,1,no,11,Spreadsheets / no tool H-1321,1,no,15,Crestline (free tier) H-1322,2-5,no,55,Business closed H-1323,1,no,12,Spreadsheets / no tool H-1324,6-20,no,166,Crestline (free tier) H-1325,2-5,no,67,Business closed H-1326,1,no,16,Crestline (free tier) H-1327,1,no,15,Crestline (free tier) H-1328,1,no,13,Crestline (free tier) H-1329,1,no,17,Crestline (free tier) H-1330,1,no,12,Crestline (free tier) H-1331,1,no,13,Crestline (free tier) H-1332,2-5,no,68,Crestline (free tier) H-1333,1,no,14,Spreadsheets / no tool H-1334,1,no,18,Crestline (free tier) H-1335,1,no,9,Crestline (free tier) H-1336,1,no,14,Crestline (free tier) H-1337,1,no,13,Crestline (free tier) H-1338,1,no,10,Crestline (free tier) H-1339,1,no,18,Spreadsheets / no tool H-1340,2-5,no,70,Crestline (free tier) H-1341,1,no,13,Spreadsheets / no tool H-1342,2-5,no,34,Crestline (free tier) H-1343,1,no,11,Crestline (free tier) H-1344,1,no,17,Spreadsheets / no tool H-1345,6-20,no,159,Crestline (free tier) H-1346,6-20,no,214,Crestline (free tier) H-1347,2-5,no,67,Crestline (free tier) H-1348,2-5,no,40,Crestline (free tier) H-1349,1,no,18,Crestline (free tier) H-1350,1,no,17,Crestline (free tier) H-1351,1,no,11,Crestline (free tier) H-1352,1,no,17,Crestline (free tier) H-1353,2-5,no,48,Business closed H-1354,1,no,12,Crestline (free tier) H-1355,1,no,16,Crestline (free tier) H-1356,1,no,14,Crestline (free tier) H-1357,1,no,18,Crestline (free tier) H-1358,1,no,11,Crestline (free tier) H-1359,1,no,17,Spreadsheets / no tool H-1360,1,no,12,Spreadsheets / no tool H-1361,2-5,no,39,Business closed H-1362,1,no,18,Crestline (free tier) H-1363,2-5,no,39,Crestline (free tier) H-1364,2-5,no,41,Business closed H-1365,2-5,no,51,Crestline (free tier) H-1366,1,no,11,Crestline (free tier) H-1367,2-5,no,76,Business closed H-1368,1,no,11,Crestline (free tier) H-1369,1,no,18,Crestline (free tier) H-1370,2-5,no,37,Business closed H-1371,1,no,9,Crestline (free tier) H-1372,1,no,18,Crestline (free tier) H-1373,2-5,no,36,Crestline (free tier) H-1374,2-5,no,77,Crestline (free tier) H-1375,2-5,no,64,Crestline (free tier) H-1376,1,no,9,Crestline (free tier) H-1377,2-5,no,71,Crestline (free tier) H-1378,1,no,10,Crestline (free tier) H-1379,1,no,12,Spreadsheets / no tool H-1380,6-20,yes,235,Business closed H-1381,1,no,9,Crestline (free tier) H-1382,1,no,15,Spreadsheets / no tool H-1383,2-5,yes,65,Crestline (free tier) H-1384,1,no,16,Crestline (free tier) H-1385,1,no,17,Spreadsheets / no tool H-1386,1,no,13,Spreadsheets / no tool H-1387,2-5,no,66,Business closed H-1388,1,no,17,Crestline (free tier) H-1389,1,no,18,Crestline (free tier) H-1390,1,no,16,Crestline (free tier) H-1391,1,no,18,Spreadsheets / no tool H-1392,1,no,15,Crestline (free tier) H-1393,1,no,10,Crestline (free tier) H-1394,1,no,11,Crestline (free tier) H-1395,2-5,no,46,Business closed H-1396,1,no,16,Crestline (free tier) H-1397,1,no,15,Crestline (free tier) H-1398,2-5,no,34,Crestline (free tier) H-1399,1,no,13,Crestline (free tier) H-1400,1,no,13,Crestline (free tier) H-1401,1,no,10,Crestline (free tier) H-1402,1,no,11,Crestline (free tier) H-1403,2-5,no,49,Crestline (free tier) H-1404,2-5,no,58,Crestline (free tier) H-1405,1,no,17,Crestline (free tier) H-1406,1,no,17,Crestline (free tier) H-1407,1,no,12,Crestline (free tier) H-1408,2-5,no,37,Crestline (free tier) H-1409,1,no,15,Spreadsheets / no tool H-1410,2-5,no,31,Crestline (free tier) H-1411,1,no,13,Spreadsheets / no tool H-1412,1,no,15,Crestline (free tier) H-1413,1,no,13,Crestline (free tier) H-1414,1,no,15,Crestline (free tier) H-1415,1,no,18,Crestline (free tier) H-1416,1,no,14,Crestline (free tier) H-1417,1,no,11,Crestline (free tier) H-1418,6-20,no,131,Crestline (free tier) H-1419,1,no,17,Crestline (free tier) H-1420,1,no,18,Crestline (free tier) H-1421,1,no,16,Crestline (free tier) H-1422,1,no,11,Crestline (free tier) H-1423,2-5,no,78,Business closed H-1424,1,no,16,Spreadsheets / no tool H-1425,1,no,18,Spreadsheets / no tool H-1426,1,no,18,Crestline (free tier) H-1427,1,no,15,Crestline (free tier) H-1428,1,no,14,Spreadsheets / no tool H-1429,1,no,14,Spreadsheets / no tool H-1430,2-5,no,63,Business closed H-1431,1,no,18,Crestline (free tier) H-1432,1,no,15,Crestline (free tier) H-1433,1,no,17,Spreadsheets / no tool H-1434,1,no,14,Crestline (free tier) H-1435,1,no,14,Crestline (free tier) H-1436,1,no,12,Crestline (free tier) H-1437,1,no,11,Crestline (free tier) H-1438,1,no,13,Crestline (free tier) H-1439,1,no,13,Spreadsheets / no tool H-1440,2-5,no,54,Business closed H-1441,1,no,14,Crestline (free tier) H-1442,1,no,16,Crestline (free tier) H-1443,2-5,no,74,Crestline (free tier) H-1444,1,no,14,Crestline (free tier) H-1445,2-5,no,65,Crestline (free tier) H-1446,1,no,18,Crestline (free tier) H-1447,1,no,11,Crestline (free tier) H-1448,1,no,10,Crestline (free tier) H-1449,2-5,no,69,Business closed
cancellation_summary.md1 lines · Download
# Cancellations, August to September: 250 accounts, about $8,100 of MRR. Before Crestline's launch we lost about 95 accounts a month.
crestline_public_info.md6 lines · Download
# Crestline (news and public pages)

- 1 August: raised $120M. 15 August: launched a free plan for 1 user (invoicing, expenses, bank feeds).
- Paid plans from $25/month; no accounting-firm partner programme.
- Open roles (September): 14 engineers for payroll; 6 for "mid-market multi-entity"; 3 for "accountant partnerships (exploring)".
- Pricing page: free plan "free forever for solo businesses".
What a strong answer doesThe answer key the graders mark against

Shows from the cancellations that Crestline is taking single-user accounts not connected to an accounting firm: 186 of the 250 cancellations went to Crestline, 142 of them single-user accounts at $9 to $19 a month, and only 4 of the 186 were connected to an accounting firm; the $5,600 of MRR lost to Crestline is a small share of a base where 1-user accounts already churned at 31% a year. The partner network is the advantage Crestline can't easily copy: 81% of 6-20 user accounts and 46% of 2-5 user accounts are connected to a firm, and they barely leave; Crestline has no partner programme. Free and price are easy to copy, and so is matching features. Reads Crestline's hiring (payroll, mid-market multi-entity, accountant partnerships 'exploring') as heading upmarket and possibly at the partner channel, the real future threat. Recommends against a free plan for everyone: it would give away revenue from 11,400 single-user accounts paying about $14 a month (about $160,000 MRR) to defend against losses that are a fraction of that. Instead: let the least valuable solo accounts go or offer a narrow solo option tied to an accounting firm, deepen the partner network (bring firms' clients onto Harbourline, lock in the firm workflow), and get ahead on payroll and multi-entity where Crestline is heading. Sets the signals that would change the call (losses spreading to connected or multi-user accounts, or Crestline launching a partner programme).

Critical failures (cap the score at 40)
  • Recommends launching a free plan for every account to match Crestline
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Needs more than a quick edit4 of 11 checks passed

The analysis contains multiple factual errors in the cancellation counts and an invented partner-firm number, so a PM would need to re-check all evidence and redo the reasoning.

Output

MEMORANDUM

TO: Board Chair FROM: VP of Product, Harbourline DATE: October 2, 2023 SUBJECT: Competitive Analysis & Strategic Response to Crestline’s Free Plan Launch

---

Executive Summary

You asked whether Crestline’s new free plan is an existential threat and whether we should launch a matching free plan for everyone.

The short answer: Crestline’s free tier is an acute hazard for our solo business segment, but it is not an existential threat to Harbourline as a whole. Launching a free plan for everyone would be a strategic error that burns revenue without neutralizing Crestline’s core advantages. Instead, we must defend our high-value accounts via our proprietary partner network, accept natural churn in the solo tier, and accelerate feature development in multi-user segments where Crestline cannot follow today.

---

1. Where Crestline Can Hurt Us (and Where It Can’t)

To understand the damage, we analyzed every account cancellation since Crestline launched their free tier on August 15 through the end of September.

  • The Damage: Total cancellations spiked to 250 accounts and ~$8,100 in MRR over August and September, compared to our historical baseline of ~95 accounts per month.
  • The Culprit: Of the 250 cancellations, 187 accounts (74.8%) explicitly migrated to Crestline’s free tier. (The remaining 63 accounts cited closed businesses or reverted to spreadsheets).

#### Where they are hurting us: Solo Businesses (1-User Accounts) Crestline’s free tier is aimed squarely at solo operators (invoicing, expenses, bank feeds for 1 user). * The Evidence: Of the 187 accounts that defected to Crestline, 164 were 1-user accounts. * Historically, our 1-user segment had an annualized churn rate of 31%—already our highest. Crestline has weaponized price against this price-sensitive, feature-light cohort. Because solo businesses rarely collaborate with an external CPA inside software (only 9% of our 1-user accounts are connected to an accounting firm), they have zero switching costs to a free tool.

#### Where they CANNOT hurt us: Multi-User & Accountant-Connected Accounts Crestline cannot effectively penetrate our core revenue engines: 1. The Partner Network Moat: Crestline has no accounting-firm partner programme. Accounts connected to our 2,100-firm partner network experience near-zero churn because Harbourline is embedded in their CPA's workflow. * The Evidence: Out of 250 cancellations in August/September, only 3 accounts connected to an accounting firm left for Crestline. 2. Multi-User Collaboration: Crestline’s free tier is strictly capped at 1 user. Businesses with 2–5 users ($52/mo average revenue) and 6–20 users ($170/mo average revenue) require team access, role-based permissions, and multi-entity tracking—none of which are supported on Crestline’s solo free tier. * While a small handful of 2–5 and 6–20 user accounts downgraded or split into solo accounts to use Crestline, the vast majority of our multi-user base is completely insulated.

---

2. What Advantages Crestline Can and Cannot Copy

With $120M in fresh capital, Crestline is executing a classic land-grab play. We must be clear-eyed about what they can replicate and what remains defensible.

#### What Crestline CAN copy: * Feature Parity for Solo Tools: Basic invoicing, expense tracking, and bank feeds are commoditized. Crestline already matches our core solo functionality. * Aggressive Go-To-Market Spend: Backed by $120M, they can outspend us on top-of-funnel acquisition for solo founders indefinitely.

#### What Crestline CANNOT copy (Our Defensible Moats): * The Accounting Firm Ecosystem (The Partner Network): Accounting practices do not adopt software one-off; they standardize their practice around shared multi-client dashboards. Crestline currently has 3 open roles exploring accountant partnerships, but building a trusted network of 2,100 firms takes years of relationship-building, workflow integration, and reputation. They cannot buy this overnight. * Embedded Collaborative Bookkeeping: For 2-to-20 user businesses, accounting is a team sport involving internal bookkeepers, business owners, and external CPAs. Harbourline’s architecture bridges this operational triad. Crestline’s solo-first roots make re-architecting for mid-market, multi-entity collaboration a heavy, multi-year engineering lift.

---

3. Where Crestline is Heading

Public job openings and product signals from September reveal Crestline’s roadmap: 1. Moving Upmarket via Payroll & Multi-Entity: They have 14 engineers dedicated to payroll and 6 building "mid-market multi-entity" features. They are using the free solo tier as a top-of-funnel trap to acquire brand awareness, intending to upsell those solo businesses if and when they grow into complex, multi-user companies. 2. The Accountant Play: Their 3 open roles for "accountant partnerships" prove they recognize their blind spot. However, going from zero to a trusted partner network is their steepest hill.

---

4. Strategic Recommendations: Should We Launch a Free Plan?

Recommendation: Do NOT launch a free plan for everyone.

Matching Crestline’s free tier across our entire customer base would be catastrophic: * Revenue Cannibalization: Harbourline generates strong revenue across 4,900 mid-sized (2-5 user) accounts and 1,300 larger (6-20 user) accounts. Giving away our product to teams that are already willing and able to pay $52 to $170/month would instantly vaporize millions in ARR. * Support Cost Bloat: Free tiers attract low-intent, high-maintenance users who strain customer support without contributing to top-line growth.

Instead, we should execute a targeted, three-part defense:

#### 1. Let Go of Unprofitable Solo Churn Accept that solo businesses with no accounting-firm connection are a low-margin acquisition battleground. Do not burn capital trying to win back price-sensitive solo operators who value $0 over workflow integration. Our blended ARPU and LTV are protected by focusing upstream.

#### 2. Deepen the Partner Network Moat (Offensive Defense) Our 2,100 accounting firm partners are our ultimate shield. We should: * Introduce incentives for accounting partners to migrate their solo clients onto Harbourline (bringing them into the network, where churn drops to near zero). * Build exclusive collaboration features between firms and multi-user clients that Crestline’s solo tool cannot touch.

#### 3. Accelerate Multi-User & Payroll Roadmap Crestline is moving upmarket; we must fortify our position there first. * Double down on team permissions, audit trails, and multi-entity reporting for our 2–5 and 6–20 user cohorts. * Ensure our payroll and expense workflows are robust enough that growing businesses have zero incentive to graduate to Crestline, but rather graduate deeper into Harbourline.

Conclusion

Crestline’s free tier is an expensive customer-acquisition engine designed to capture the lowest tier of the market. By holding our pricing power, protecting our accounting partner network, and doubling down on multi-user collaboration, we will cede low-value churn while retaining the high-value businesses that drive our enterprise value.

Claims the judge couldn’t find in the brief

  • Of the 250 cancellations, 187 accounts (74.8%) explicitly migrated to Crestline’s free tier.
  • Of the 187 accounts that defected to Crestline, 164 were 1-user accounts.
  • only 3 accounts connected to an accounting firm left for Crestline.
  • 2,100-firm partner network
  • Accounts connected to our 2,100-firm partner network experience near-zero churn because Harbourline is embedded in their CPA's workflow.

Grades and run details

Decision model 64 · LLM judge 33
Decision model checks
  • failUses the supplied evidence correctly34%
  • partialAddresses the actual decision47%
  • passRespects explicit constraints54%
  • partialIdentifies material uncertainty52%
  • partialAvoids unsupported claims26%
  • passProduces the required deliverable76%
  • partialFinds who's actually leaving48%
  • passReads where Crestline is heading100%
  • passCompetes with what customers really choose40%
  • failGets the base of every number right10%
  • passSeparates what a rival can copy100%
Run
Run
#1
Time to output
9 s
Submitted
5 Oct 2026

Usable with a quick edit9 of 11 checks passed

The analysis is evidence-based, logically structured, and directly answers the board's question; a product manager could use it with minor wording tweaks and no rework of evidence or reasoning.

Output

Board memo: Crestline’s free plan—targeted threat, not a case for universal free

Recommendation: Do not launch a free plan for everyone. Crestline threatens customers buying standalone bookkeeping, including some larger businesses—not just solos. Our strongest protection is the accounting-firm relationship embedded in Harbourline, not account size or a feature lead. We should test a tightly bounded free offer, defend unconnected customers across sizes, and deepen our partner advantage before Crestline builds its own.

1. Where Crestline can hurt us—and where it is not yet winning

The threat is real, but headline cancellations overstate incremental damage

The cancellation file totals 250 accounts and $8,065 in lost MRR. Of these:

  • 186 accounts, representing $5,581 MRR, went to Crestline.
  • 38 returned to spreadsheets/no tool; 26 closed their businesses.
  • Crestline-associated losses equal approximately 0.88% of our current $635,400 MRR.

Against the previous run rate of 95 cancellations monthly, 250 over August–September is about 60 additional cancellations—not 186. Destination tells us where customers went, not whether they would otherwise have stayed. The mid-August launch also complicates the comparison.

This is a meaningful warning, not evidence of an existential threat. Equally, cancellations do not capture prospects we failed to acquire or future pricing pressure.

Accounting-firm connection explains exposure better than size

Calculated from `cancellationsaugsep.csv` and `customer_base.md`:

Account sizeUnconnected exits to Crestline / estimated unconnected baseConnected exits to Crestline / estimated connected baseMRR lost to Crestline
------:---:---:
1 user142 / 10,374 (1.37%)0 / 1,026 (0%)$1,958
2–5 users31 / 2,646 (1.17%)3 / 2,254 (0.13%)$1,850
6–20 users9 / 247 (3.64%)1 / 1,053 (0.09%)$1,773

These are observed-period exit ratios using September-end balances and reported connection shares, not cohort churn rates.

Where Crestline can hurt us:

  • Standalone solos: They account for 142 of 186 competitive exits. Free invoicing, expenses and bank feeds directly challenge a $14/month product. This segment already had 31% annual churn before launch.
  • Unconnected larger businesses: These are not protected simply because they have more users. The highest observed competitive exit ratio is among unconnected 6–20-user accounts, albeit from only nine exits.
  • Revenue beyond solos: Multi-user customers represent only 24% of Crestline exits but 65% of the associated lost MRR. A solo-only defensive strategy would miss most of the revenue leakage.

There is an important inconsistency to investigate: 44 multi-user Harbourline accounts report moving to a one-user free plan. They may have reduced usage, retained other tools, or reported their destination imprecisely. We should verify their actual workflows and Crestline plans before claiming full multi-user substitution. The revenue loss is nevertheless real.

Where Crestline is not yet winning materially:

Only four accounting-firm-connected accounts moved to Crestline, versus 182 unconnected accounts. Relative to estimated bases, that is approximately 0.09% versus 1.37%. Connected customers account for roughly 49% of our MRR.

An accountant reconciles, files and advises inside Harbourline. Replacing us therefore means changing a shared operating workflow, not just choosing cheaper software. Crestline currently has no accounting-firm partner programme. That is strong evidence of protection today—not immunity: four connected customers did leave, and customer selection may explain some of the retention difference.

2. Which advantages can it copy?

AdvantageAssessment
Core bookkeeping, invoicing, expenses and bank-feed functionalityCopyable; partly already matched. These are not a sufficient basis for defending subscription pricing.
Collaboration features and accountant-facing softwareCopyable with investment. The $120M raise gives Crestline resources, although funding alone does not prove execution.
Larger-business functionalityContestable. Its hiring signals an intention to close gaps; size is not a durable barrier.
Our 2,100-firm network and embedded client–accountant workflowsNot quickly reproducible by shipping features. Crestline must recruit firms, earn trust, change working practices and migrate client relationships. It can build an alternative over time, but cannot instantly copy our existing relationships.

Our moat is therefore the working network, not merely the existence of an accountant portal. We should strengthen productive firm–customer connections rather than assume the partner count itself guarantees defensibility. We have no evidence here for additional advantages in brand, data, accuracy or cost.

3. Where Crestline is heading

The public hiring mix is more informative than “free forever” alone:

  • 14 payroll engineers: a substantial signal toward an adjacent recurring workflow.
  • Six mid-market multi-entity engineers: evidence of ambition beyond solos.
  • Three accountant-partnership roles, labelled “exploring”: early recognition of the channel where we are strongest.

The likely strategy is free acquisition at the bottom, paid expansion through greater complexity and adjacent products. Paid plans already start at $25/month. Payroll and multi-entity could create monetisation opportunities while preserving the solo-free promise.

This is a directional inference, not a confirmed roadmap. Hiring does not establish launch dates or product quality. Nevertheless, we should expect a broader competitor, not permanently dismiss Crestline as a free solo tool.

4. What we should do

1. Reject universal free; authorise a controlled test. Our solo segment alone generates approximately $159,600 MRR, versus $1,958 lost to Crestline in the file. Those figures are not a forecast, but show the cannibalisation risk. Test a limited one-user offer for new self-serve prospects and selected at-risk customers, with clear usage boundaries and paid upgrade paths. Use holdouts to measure incremental activation, retained contribution, conversion, support costs and paid-plan downgrades—not free registrations.

2. Defend unconnected customers across all sizes immediately. Interview the multi-user switchers first. Establish whether price, unused seats, workflow simplicity or missing value drove departure. Improve onboarding and retention offers accordingly; do not default to discounts for every customer.

3. Invest primarily in the partner workflow. Make firm connection, shared reconciliation, filing handoffs and multi-client work easier. Help existing firms connect more suitable clients. Measure active shared workflows and cohort retention, not merely partner sign-ups.

4. Review in 90 days against explicit evidence. Track competitive losses and acquisition win rates by size and firm connection, lost MRR, partner recruitment and Crestline’s product releases. Reconcile the historical annual-churn figures with the separate 95-per-month baseline before making causal claims.

Decision requested: Approve a bounded free-plan experiment and partner-led defence—not a universal price reset. Reconsider broader action if connected-customer losses rise materially or controlled tests show that free creates sustainable incremental value.

Grades and run details

Decision model 82 · LLM judge 100
Decision model checks
  • failUses the supplied evidence correctly69%
  • passAddresses the actual decision91%
  • passRespects explicit constraints42%
  • passIdentifies material uncertainty90%
  • passAvoids unsupported claims40%
  • passProduces the required deliverable76%
  • passFinds who's actually leaving84%
  • passReads where Crestline is heading100%
  • passCompetes with what customers really choose73%
  • failGets the base of every number right59%
  • passSeparates what a rival can copy100%
Run
Run
#1
Time to output
87 s
Submitted
5 Oct 2026

Usable with a quick edit9 of 11 checks passed

The analysis is evidence-based, actionable, and flags its own data caveats; a PM could use it with only minor wording tweaks.

Output

Crestline: Competitive Analysis for the Board

From: VP Product | Re: Board chair's question on a free plan

Summary

Crestline is not an existential threat today. In two months it took 186 accounts and about $5,600 of MRR, which is 0.9% of our roughly $635k MRR. Almost all of those accounts were the ones least tied to us: single users not connected to an accounting firm.

It has barely touched accounts that work with their accountant inside Harbourline. That network is the one advantage Crestline cannot quickly buy.

The real risk is where Crestline is heading. It is building payroll and multi-entity features, and it has started on accountant partnerships. It also already appears to be winning our unconnected larger accounts.

My recommendation is not to launch a free plan for everyone. It would put up to our entire MRR at risk of downgrade to counter a loss of under 1%. Instead, we should spend the next two quarters getting unconnected teams connected to firms, locking in our firm network, and closing the payroll gap.

What the cancellations show

These figures come from all 250 cancellations in August and September.

DestinationAccountsMRR lost
Crestline186$5,581
Business closed26$1,929
Spreadsheets / no tool38$555
Total250$8,065

Losses to Crestline by segment and firm connection

SegmentUnconnected baseLost to Crestline2-month rateConnected baseLost to Crestline
1 user~10,3701421.4%~1,0300
2–5 users~2,650311.2%~2,2503
6–20 users~25093.6%~1,0501
Total~13,2701821.4%~4,3304 (0.09%)

Where Crestline can hurt us

1. Unconnected solo businesses. This is the main battleground so far: 142 of the 186 accounts we lost to Crestline. These accounts pay $14 a month and already churned 31% a year before Crestline launched. We are losing them, but they were always our least durable revenue. Another 38 solo accounts left for spreadsheets, which shows that some of this segment simply doesn't value paid software.

2. Unconnected larger accounts. This is the warning sign. Nine of roughly 250 unconnected 6–20 user accounts went to Crestline in two months, about 3.6%. Annualised, that is above 20%, against a historical churn of 6% for the whole segment. These accounts pay around $170 a month.

Crestline's free plan covers only one user. So these teams either moved to Crestline's paid plans (from $25 a month) or cut down to one user. Either way, Crestline is competing on product and price for teams, not just on "free."

The numbers are small, but this pool is worth about $42k of MRR. Unconnected 2–5 user accounts are worth about $138k and are losing about 1.2% to Crestline every two months.

Where Crestline can't hurt us (yet)

Accounts connected to an accounting firm. Only 4 of about 4,330 connected accounts went to Crestline. That is roughly 0.09%, about 15 times lower than unconnected accounts.

Of the 8 connected accounts that cancelled at all, half did so because the business closed. Connected accounts make up about half our MRR (roughly $310k). Most of that is in the 2–5 and 6–20 user segments, where 46% and 81% of accounts are connected.

There is a caveat. Connection may partly reflect businesses that were already stickier, so the 15x gap is not proof that connecting an account causes retention. Still, Crestline has no partner programme. A firm that runs many clients inside Harbourline would have to move its whole practice to switch, which makes these accounts very costly to pull away.

Which advantages it could copy, and which it couldn't

Easily copied, or already matched: - Invoicing, expenses and bank feeds (already in Crestline's free plan) - Price - Payroll: 14 engineers are hiring for it, and we have no equivalent advantage - Multi-entity features for larger customers

Hard to copy: - Our 2,100-firm partner network - The fact that firms reconcile, file and advise inside Harbourline. That is a workflow habit and a two-sided network, not a feature.

Crestline has 3 open roles for accountant partnerships, labelled "exploring." With $120M it could offer firms rich incentives such as revenue share and free practice tools. But recruiting firms one at a time and changing how they work takes years, not quarters. Our advantage is durable, not permanent. Our window is now, while Crestline's effort is still exploratory.

Where Crestline is heading

Its September hiring is the clearest signal: - Payroll (14 engineers). This is its biggest bet. A bundled payroll offer would hit our 2–5 and 6–20 user segments directly. - Mid-market multi-entity (6 engineers). This moves it upmarket into the segment where we earn the most per account. - Accountant partnerships (3, exploring). This is the move against our network. It is early, but it is aimed at the right target.

The most likely pattern: free solo plans bring in new businesses cheaply, then Crestline moves those customers onto paid plans as they grow. It uses payroll and multi-entity features to win teams, and eventually goes after the firms.

Recommendations

1. Don't launch a free plan for everyone. Crestline doesn't offer one either, because its free plan is solo only. A free plan for everyone would put our ~$635k MRR at risk of downgrade to counter ~$5.6k lost over two months.

Even matching Crestline's solo-only free plan exposes about $160k of MRR (~$1.9M a year). If just 10% of solo accounts downgraded, we would lose more each month than Crestline has taken from us in total.

2. Use the firm network to defend unconnected teams. About 2,900 multi-user accounts are not connected to a firm. We should test a programme that introduces them to partner firms, for example a matching service, a free first review, or a short discount once connected.

We should measure whether connecting accounts actually reduces churn before rolling it out widely. If it does, this is the highest-return action available to us.

3. Lock in our firms before Crestline arrives. We should offer firms better tools for managing their practice, rewards for bringing clients onto Harbourline, and multi-year partner terms. Our 2,100 firms should have a reason to stay that goes beyond habit.

4. Close the payroll gap. Within a quarter, decide whether to build payroll or partner for it. If Crestline ships payroll first to our teams, our current advantage weakens.

5. Compete for solos through firms, not on price. One option is a low-cost or free solo plan available only when a partner firm sponsors or connects the client. This turns our advantage into a way to win new customers without devaluing our paid plans. We should accept some loss of unconnected solo accounts for now.

6. Set triggers to revisit this decision. Escalate to the board if any of the following happens: - Losses of connected accounts to Crestline exceed 0.5% per quarter - Losses of unconnected 6–20 user accounts stay above 3% per quarter - Crestline announces a formal accountant programme

Data issues to resolve

The baseline churn figures disagree. The summary says we lost about 95 accounts a month before Crestline launched. But the annual churn rates by segment imply roughly 350 a month. On the first figure, August–September is about 60 accounts above normal. On the second, it is below normal. Finance needs to reconcile these before we report a net Crestline impact.

The destination label looks wrong for multi-user accounts. The cancellation file records 44 multi-user accounts as going to "Crestline (free tier)," which only allows one user. We should confirm through exit interviews whether they went to paid plans or reduced to one user, because that changes how we respond.

The time window is mixed. The window includes the first half of August, before Crestline's free plan launched on 15 August, so the post-launch picture is slightly blended.

Conclusion

Crestline is winning the customers we hold most loosely, and it is investing in the areas where we are vulnerable. A free plan would trade our strongest economics for our weakest segment. The better response is to deepen the firm network, which Crestline cannot buy quickly, and to close the payroll gap before it arrives.

Grades and run details

Decision model 77 · LLM judge 100
Decision model checks
  • failUses the supplied evidence correctly35%
  • passAddresses the actual decision100%
  • passRespects explicit constraints27%
  • passIdentifies material uncertainty99%
  • partialAvoids unsupported claims55%
  • passProduces the required deliverable80%
  • passFinds who's actually leaving96%
  • passReads where Crestline is heading100%
  • passCompetes with what customers really choose95%
  • failGets the base of every number right61%
  • passSeparates what a rival can copy100%
Run
Run
#1
Time to output
1.9 min
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 7

Uses the supplied evidence correctlyWrongMixedMixed
Gemini 3.5 Flash-Lite · Gemini

Multiple factual errors: Crestline defectors count is 186 not 187, 1-user defectors 142 not 164, connected-account defectors 4 not 3, and 2,100 firms is invented.

GPT-6 Astra · ChatGPT

All factual claims about the current situation are drawn directly from the supplied context or computed correctly from it; no invented facts.

Opus 5.5 · Claude

All factual claims about the current situation are directly supported by the supplied context or arithmetic from it, with no invented facts.

Addresses the actual decisionWrongRightRight
Gemini 3.5 Flash-Lite · Gemini

The output commits to a clear answer (do not launch a free plan) but does not state what result or condition would change that call.

GPT-6 Astra · ChatGPT

Commits early to 'Do not launch a free plan for everyone' and states that rising connected-customer losses or controlled-test results would change the call.

Opus 5.5 · Claude

The output clearly recommends against a free plan for everyone and specifies triggers (loss thresholds, Crestline accountant programme) that would change the call.

Identifies material uncertaintyWrongRightRight
Gemini 3.5 Flash-Lite · Gemini

The output does not name specific unknowns that could change the decision or say how they would be resolved.

GPT-6 Astra · ChatGPT

Identifies the multi-user switcher inconsistency, the limits of hiring signals, and specifies the evidence (connected losses, test results) that would change the decision.

Opus 5.5 · Claude

It names specific unknowns (baseline churn discrepancy, multi-user destination label, pre-launch blending) and says how to resolve them, plus sets escalation triggers.

Avoids unsupported claimsWrongRightRight
Gemini 3.5 Flash-Lite · Gemini

Interpretations like 'expensive customer-acquisition engine' and 'heavy, multi-year engineering lift' are presented as fact without hedging.

GPT-6 Astra · ChatGPT

Forecasts and interpretations are labelled as such (e.g., 'likely strategy', 'directional inference'), and confident claims are backed by the data.

Opus 5.5 · Claude

Interpretations and forecasts are labelled as such (e.g., 'appears to be winning', 'most likely pattern'), and confident claims are backed by evidence.

Finds who's actually leavingWrongRightRight
Gemini 3.5 Flash-Lite · Gemini

The analysis miscounts the cancellations: 1-user defectors are 142, not 164, and connected-account defectors are 4, not 3, so the sizing is inaccurate.

GPT-6 Astra · ChatGPT

Counts cancellations by size and partner connection, sizes the MRR lost, and sets it against the base, showing losses are overwhelmingly unconnected single-user accounts.

Opus 5.5 · Claude

It breaks down cancellations by segment and firm connection, shows 142 single-user unconnected lost to Crestline, sizes the MRR, and sets it against the base.

Competes with what customers really chooseMixedRightRight
Gemini 3.5 Flash-Lite · Gemini

The output mentions spreadsheets/no tool as a destination but does not analyze the status quo as a competitor or draw a direction-of-travel insight from it.

GPT-6 Astra · ChatGPT

Treats spreadsheets/no tool as a competitor, notes the status quo, and draws a direction-of-travel insight from Crestline's investment signals.

Opus 5.5 · Claude

It treats spreadsheets/no-tool as a competitor, notes that some solos don't value paid software, and draws implications from Crestline's investment direction.

Gets the base of every number rightWrongMixedMixed
Gemini 3.5 Flash-Lite · Gemini

Percentages and counts are wrong: 74.8% should be 74.4%, 164 should be 142, 3 should be 4, and the 2,100 firms base is unsupported.

GPT-6 Astra · ChatGPT

All percentages and differences use the correct denominators (e.g., unconnected/connected bases, total MRR) and the bases are stated where ambiguous.

Opus 5.5 · Claude

All derived percentages and MRR figures check out against the supplied data, with bases stated where needed (e.g., unconnected base sizes).

All got right 4

Respects explicit constraintsRightRightRight
Gemini 3.5 Flash-Lite · Gemini

The output is a memo to the board chair, within the 1,200-word limit, and addresses the requested competitive analysis.

GPT-6 Astra · ChatGPT

Delivers a board memo under 1,200 words, addresses the board chair's question, and proposes actions that would enforce the recommendation.

Opus 5.5 · Claude

The output is a competitive analysis under 1,200 words, addressed to the board, and respects all stated constraints.

Produces the required deliverableRightRightRight
Gemini 3.5 Flash-Lite · Gemini

The memo is complete, in the right form for the board chair, within length, and could be acted on with light edits.

GPT-6 Astra · ChatGPT

The memo is complete, in the right form for the board, within the word limit, and actionable with light edits.

Opus 5.5 · Claude

The memo is a complete competitive analysis with summary, evidence, and recommendations, ready for the board with at most light edits.

Reads where Crestline is headingRightRightRight
Gemini 3.5 Flash-Lite · Gemini

The output correctly reads Crestline's hiring as moving toward payroll, mid-market multi-entity, and exploring accountant partnerships, and flags the partner-programme risk.

GPT-6 Astra · ChatGPT

Uses Crestline's hiring (payroll, multi-entity, accountant partnerships) to infer direction, flags the partner-programme risk, and adjusts the response accordingly.

Opus 5.5 · Claude

It reads Crestline's hiring (payroll, multi-entity, accountant partnerships) as direction, flags the partner-programme risk, and adjusts the response accordingly.

Separates what a rival can copyRightRightRight
Gemini 3.5 Flash-Lite · Gemini

The output separates copyable advantages (features, price, spend) from durable ones (partner network, embedded collaboration) and anchors the response in the latter.

GPT-6 Astra · ChatGPT

Separates copyable advantages (features, price) from the durable partner network and embedded workflows, and anchors the response in the latter.

Opus 5.5 · Claude

It separates easily copied advantages (features, price) from durable ones (partner network, workflow habits) and anchors the response in the durable ones.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Opus 5.5withClaude86.4100.02None
2GPT-6 AstrawithChatGPT88.695.82None
3GPT-6.1 SolwithAPI88.695.82None
4GPT-6 LunawithAPI93.283.32None
5Sonnet 5.5withAPI86.483.32None
6Haiku 5.5withAPI84.179.22None
7Gemini 3.8 FlashwithAPI68.254.22None
8Gemini 3.5 Flash-LitewithGemini61.425.02None

What you get for the cost

GPT-6.1 Sol · API scores within a point of Opus 5.5 · Claude at a fifth of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreOpus 5.5 · Claude: task score 94, 75× the cheapestGPT-6 Astra · ChatGPT: task score 93, 87× the cheapestGPT-6.1 Sol · API: task score 93, 14× the cheapestGPT-6 Luna · API: task score 86, the cheapestSonnet 5.5 · API: task score 83, 44× the cheapestHaiku 5.5 · API: task score 80, 3.0× the cheapestGemini 3.8 Flash · API: task score 60, 12× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 41, 2.9× the cheapestOpus 5.5 · ClaudeGPT-6 Astra · ChatGPTGPT-6.1 Sol · APIGPT-6 Luna · APISonnet 5.5 · APIHaiku 5.5 · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and cost
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API861.0×
Gemini 3.5 Flash-Lite · Gemini412.9×
Haiku 5.5 · API803.0×
Gemini 3.8 Flash · API6012×
GPT-6.1 Sol · API9314×
Sonnet 5.5 · API8344×
Opus 5.5 · Claude9475×
GPT-6 Astra · ChatGPT9387×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Deciding how to respond to competitors.

Why it matters

Feature grids hide the real competition, which is often the customer's current workaround, and the response that feels urgent is usually to copy everything.

What good looks like

  • Includes the status quo and non-obvious alternatives
  • Finds where losses really go, by segment
  • Separates advantages a rival can copy from ones it can't
  • Reads where competitors are heading

Deliberately not measured

  • Market sizing
Capability tested

Competitive insight

The failure we’re looking for

A feature checklist and a plan to match everything

Grading

Decision model and LLM judge, calibrated against a blind PM review