Tasks / Strategy

Competitor analysis

Can the model find where a competitor can actually hurt you, from win/loss and churn data, rather than list features?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 63% were usable with at most a quick edit.

Reliably right

  1. Respects explicit constraints100% pass
    Delivers a competitive analysis memo to Dan within 900 words, addressing all requested topics without proposing a full feature-by-feature match.
    Opus 5.5 · Claude · The competitor isn't who you think
  2. Reads where Crestline is heading100% pass
    Uses Crestline's hiring (payroll, multi-entity, accountant partnerships) to infer direction, flags the partner-programme risk, and adjusts the response accordingly.
    GPT-6 Astra · ChatGPT · Free forever, funded
  3. Produces the required deliverable97% pass
    The memo is complete, in the right form for the CEO, within the word limit, and actionable with light edits.
    Opus 5.5 · Claude · The competitor isn't who you think

Where it slips

  1. Uses the supplied evidence correctly56% pass
    Two factual claims are wrong: the number of unconnected multi-user accounts going to Crestline is 40, not 42, and the number of connected 6-20 cancellations is 5, not 4.
    Sonnet 5.5 · API · Free forever, funded
  2. Gets the base of every number right61% pass
    Derived figures for unconnected multi-user cancellations (42 vs actual 40) and connected 6-20 cancellations (4 vs actual 5) are incorrect.
    Sonnet 5.5 · API · Free forever, funded
  3. Separates what a rival can copy66% pass
    Does not explicitly separate advantages a well-funded competitor could copy from those it couldn't, nor anchor the response in durable advantages with evidence.
    GPT-6 Luna · API · The competitor isn't who you think

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Competes with what customers really choose

    Does the analysis treat the status quo (spreadsheets, manual work, doing nothing) as a competitor, and draw at least one implication from where competitors are investing rather than only from today's features?

    Passes when Analyses the status quo, strengths included, and gives a direction-of-travel insight that affects the recommendation.

  2. Separates what a rival can copy

    Does the analysis separate advantages a well-funded competitor could copy (features, price, being first) from ones it couldn't easily (switching costs, networks, relationships), and build the response on the latter?

    Passes when Names which advantages are copyable and which aren't, with evidence, and anchors the response in the durable ones.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the VP Product at Harbourline. Crestline raised $120M in August and launched a free plan for solo businesses. Our board chair wants to know whether we should launch a free plan for everyone to match it (the note is below). Write the board a competitive analysis of no more than 1,200 words: where Crestline can hurt us and where it can't, with the evidence; which of our advantages it could copy and which it couldn't; where it's heading; and what we should do. The files are attached.

What the model was given6 items: About Harbourline, board_chair_note.md, customer_base.md, cancellations_aug_sep.csv (every account that cancelled since Crestline's launch), cancellation_summary.md, crestline_public_info.md
About HarbourlineBookkeeping, invoicing and expenses for small businesses, with a partner network that connects customers to their accounting firm.
board_chair_note.md1 lines · Download
From the board chair: "Crestline's free tier is an existential threat. Should we launch a free plan for everyone to match it? We need a view on where they can hurt us and where they can't."
customer_base.md9 lines · Download
# Harbourline customers (end of September)

| Account size | Paying accounts | Share connected to an accounting firm | Average revenue per month | Annual churn before Crestline's free tier |
|---|---|---|---|---|
| 1 user | 11,400 | 9% | $14 | 31% |
| 2-5 users | 4,900 | 46% | $52 | 14% |
| 6-20 users | 1,300 | 81% | $170 | 6% |
…
cancellations_aug_sep.csv (every account that cancelled since Crestline's launch)cancelled_account,account_users,connected_to_accounting_firm,mrr_usd,went_to H-1200,1,no,14,Crestline (free tier) H-1201,1,no,10,Crestline (free tier) H-1202,2-5,no,58,Crestline (free tier) H-1203,1,no,11,Crestline (free tier) H-1204,1,no,17,Crestline (free tier) H-1205,1,no,15,Spreadsheets / no tool H-1206,1,no,15,Spreadsheets / no tool H-1207,1,no,9,Crestline (free tier) H-1208,2-5,no,66,Crestline (free tier) H-1209,2-5,no,44,Crestline (free tier) H-1210,1,no,11,Crestline (free tier) H-1211,1,no,9,Crestline (free tier) H-1212,1,no,12,Crestline (free tier) H-1213,1,no,16,Crestline (free tier) H-1214,2-5,yes,73,Crestline (free tier) H-1215,1,no,9,Crestline (free tier) H-1216,2-5,no,56,Crestline (free tier) H-1217,1,no,14,Crestline (free tier) H-1218,2-5,no,47,Business closed H-1219,1,no,16,Crestline (free tier) H-1220,6-20,no,177,Crestline (free tier) H-1221,2-5,yes,51,Crestline (free tier) H-1222,2-5,no,54,Business closed H-1223,6-20,no,168,Crestline (free tier) H-1224,1,no,18,Crestline (free tier) H-1225,1,no,13,Spreadsheets / no tool H-1226,1,no,11,Crestline (free tier) H-1227,1,no,15,Crestline (free tier) H-1228,1,no,12,Crestline (free tier) H-1229,1,no,13,Crestline (free tier) H-1230,2-5,no,51,Business closed H-1231,1,no,18,Crestline (free tier) H-1232,1,no,16,Spreadsheets / no tool H-1233,1,no,12,Crestline (free tier) H-1234,1,no,16,Spreadsheets / no tool H-1235,1,no,12,Spreadsheets / no tool H-1236,1,no,13,Crestline (free tier) H-1237,1,no,17,Spreadsheets / no tool H-1238,6-20,yes,188,Business closed H-1239,1,no,17,Crestline (free tier) H-1240,1,no,15,Crestline (free tier) H-1241,6-20,yes,238,Crestline (free tier) H-1242,1,no,11,Spreadsheets / no tool H-1243,2-5,no,66,Crestline (free tier) H-1244,2-5,no,63,Business closed H-1245,6-20,no,199,Crestline (free tier) H-1246,1,no,18,Crestline (free tier) H-1247,1,no,11,Crestline (free tier) H-1248,1,no,15,Crestline (free tier) H-1249,2-5,no,60,Crestline (free tier) H-1250,2-5,no,66,Crestline (free tier) H-1251,2-5,no,34,Crestline (free tier) H-1252,1,no,12,Crestline (free tier) H-1253,1,no,18,Crestline (free tier) H-1254,1,no,18,Spreadsheets / no tool H-1255,2-5,no,31,Crestline (free tier) H-1256,1,no,17,Crestline (free tier) H-1257,1,no,16,Crestline (free tier) H-1258,1,no,18,Spreadsheets / no tool H-1259,1,no,9,Crestline (free tier) H-1260,1,no,15,Crestline (free tier) H-1261,1,no,18,Crestline (free tier) H-1262,6-20,yes,170,Business closed H-1263,1,no,18,Crestline (free tier) H-1264,2-5,no,41,Business closed H-1265,1,no,14,Spreadsheets / no tool H-1266,1,no,14,Crestline (free tier) H-1267,1,no,12,Crestline (free tier) H-1268,6-20,no,112,Crestline (free tier) H-1269,1,no,9,Crestline (free tier) H-1270,2-5,no,71,Business closed H-1271,1,no,12,Crestline (free tier) H-1272,1,no,11,Crestline (free tier) H-1273,1,no,15,Crestline (free tier) H-1274,1,no,9,Crestline (free tier) H-1275,6-20,no,209,Crestline (free tier) H-1276,2-5,no,36,Crestline (free tier) H-1277,1,no,13,Crestline (free tier) H-1278,1,no,11,Crestline (free tier) H-1279,1,no,15,Crestline (free tier) H-1280,1,no,18,Spreadsheets / no tool H-1281,1,no,16,Crestline (free tier) H-1282,1,no,17,Crestline (free tier) H-1283,1,no,9,Crestline (free tier) H-1284,1,no,10,Crestline (free tier) H-1285,6-20,yes,128,Business closed H-1286,1,no,17,Crestline (free tier) H-1287,1,no,12,Spreadsheets / no tool H-1288,2-5,no,39,Crestline (free tier) H-1289,1,no,14,Crestline (free tier) H-1290,1,no,12,Crestline (free tier) H-1291,1,no,16,Crestline (free tier) H-1292,2-5,no,62,Crestline (free tier) H-1293,1,no,12,Crestline (free tier) H-1294,1,no,12,Crestline (free tier) H-1295,1,no,9,Spreadsheets / no tool H-1296,1,no,11,Crestline (free tier) H-1297,1,no,16,Crestline (free tier) H-1298,1,no,17,Crestline (free tier) H-1299,1,no,13,Spreadsheets / no tool H-1300,1,no,13,Crestline (free tier) H-1301,1,no,17,Crestline (free tier) H-1302,1,no,12,Crestline (free tier) H-1303,1,no,17,Crestline (free tier) H-1304,1,no,14,Crestline (free tier) H-1305,1,no,9,Crestline (free tier) H-1306,1,no,15,Crestline (free tier) H-1307,1,no,16,Crestline (free tier) H-1308,1,no,16,Crestline (free tier) H-1309,1,no,12,Crestline (free tier) H-1310,2-5,no,47,Business closed H-1311,2-5,no,78,Crestline (free tier) H-1312,1,no,11,Crestline (free tier) H-1313,1,no,14,Crestline (free tier) H-1314,1,no,15,Spreadsheets / no tool H-1315,1,no,14,Spreadsheets / no tool H-1316,1,no,9,Crestline (free tier) H-1317,2-5,no,52,Business closed H-1318,1,no,13,Crestline (free tier) H-1319,2-5,no,43,Business closed H-1320,1,no,11,Spreadsheets / no tool H-1321,1,no,15,Crestline (free tier) H-1322,2-5,no,55,Business closed H-1323,1,no,12,Spreadsheets / no tool H-1324,6-20,no,166,Crestline (free tier) H-1325,2-5,no,67,Business closed H-1326,1,no,16,Crestline (free tier) H-1327,1,no,15,Crestline (free tier) H-1328,1,no,13,Crestline (free tier) H-1329,1,no,17,Crestline (free tier) H-1330,1,no,12,Crestline (free tier) H-1331,1,no,13,Crestline (free tier) H-1332,2-5,no,68,Crestline (free tier) H-1333,1,no,14,Spreadsheets / no tool H-1334,1,no,18,Crestline (free tier) H-1335,1,no,9,Crestline (free tier) H-1336,1,no,14,Crestline (free tier) H-1337,1,no,13,Crestline (free tier) H-1338,1,no,10,Crestline (free tier) H-1339,1,no,18,Spreadsheets / no tool H-1340,2-5,no,70,Crestline (free tier) H-1341,1,no,13,Spreadsheets / no tool H-1342,2-5,no,34,Crestline (free tier) H-1343,1,no,11,Crestline (free tier) H-1344,1,no,17,Spreadsheets / no tool H-1345,6-20,no,159,Crestline (free tier) H-1346,6-20,no,214,Crestline (free tier) H-1347,2-5,no,67,Crestline (free tier) H-1348,2-5,no,40,Crestline (free tier) H-1349,1,no,18,Crestline (free tier) H-1350,1,no,17,Crestline (free tier) H-1351,1,no,11,Crestline (free tier) H-1352,1,no,17,Crestline (free tier) H-1353,2-5,no,48,Business closed H-1354,1,no,12,Crestline (free tier) H-1355,1,no,16,Crestline (free tier) H-1356,1,no,14,Crestline (free tier) H-1357,1,no,18,Crestline (free tier) H-1358,1,no,11,Crestline (free tier) H-1359,1,no,17,Spreadsheets / no tool H-1360,1,no,12,Spreadsheets / no tool H-1361,2-5,no,39,Business closed H-1362,1,no,18,Crestline (free tier) H-1363,2-5,no,39,Crestline (free tier) H-1364,2-5,no,41,Business closed H-1365,2-5,no,51,Crestline (free tier) H-1366,1,no,11,Crestline (free tier) H-1367,2-5,no,76,Business closed H-1368,1,no,11,Crestline (free tier) H-1369,1,no,18,Crestline (free tier) H-1370,2-5,no,37,Business closed H-1371,1,no,9,Crestline (free tier) H-1372,1,no,18,Crestline (free tier) H-1373,2-5,no,36,Crestline (free tier) H-1374,2-5,no,77,Crestline (free tier) H-1375,2-5,no,64,Crestline (free tier) H-1376,1,no,9,Crestline (free tier) H-1377,2-5,no,71,Crestline (free tier) H-1378,1,no,10,Crestline (free tier) H-1379,1,no,12,Spreadsheets / no tool H-1380,6-20,yes,235,Business closed H-1381,1,no,9,Crestline (free tier) H-1382,1,no,15,Spreadsheets / no tool H-1383,2-5,yes,65,Crestline (free tier) H-1384,1,no,16,Crestline (free tier) H-1385,1,no,17,Spreadsheets / no tool H-1386,1,no,13,Spreadsheets / no tool H-1387,2-5,no,66,Business closed H-1388,1,no,17,Crestline (free tier) H-1389,1,no,18,Crestline (free tier) H-1390,1,no,16,Crestline (free tier) H-1391,1,no,18,Spreadsheets / no tool H-1392,1,no,15,Crestline (free tier) H-1393,1,no,10,Crestline (free tier) H-1394,1,no,11,Crestline (free tier) H-1395,2-5,no,46,Business closed H-1396,1,no,16,Crestline (free tier) H-1397,1,no,15,Crestline (free tier) H-1398,2-5,no,34,Crestline (free tier) H-1399,1,no,13,Crestline (free tier) H-1400,1,no,13,Crestline (free tier) H-1401,1,no,10,Crestline (free tier) H-1402,1,no,11,Crestline (free tier) H-1403,2-5,no,49,Crestline (free tier) H-1404,2-5,no,58,Crestline (free tier) H-1405,1,no,17,Crestline (free tier) H-1406,1,no,17,Crestline (free tier) H-1407,1,no,12,Crestline (free tier) H-1408,2-5,no,37,Crestline (free tier) H-1409,1,no,15,Spreadsheets / no tool H-1410,2-5,no,31,Crestline (free tier) H-1411,1,no,13,Spreadsheets / no tool H-1412,1,no,15,Crestline (free tier) H-1413,1,no,13,Crestline (free tier) H-1414,1,no,15,Crestline (free tier) H-1415,1,no,18,Crestline (free tier) H-1416,1,no,14,Crestline (free tier) H-1417,1,no,11,Crestline (free tier) H-1418,6-20,no,131,Crestline (free tier) H-1419,1,no,17,Crestline (free tier) H-1420,1,no,18,Crestline (free tier) H-1421,1,no,16,Crestline (free tier) H-1422,1,no,11,Crestline (free tier) H-1423,2-5,no,78,Business closed H-1424,1,no,16,Spreadsheets / no tool H-1425,1,no,18,Spreadsheets / no tool H-1426,1,no,18,Crestline (free tier) H-1427,1,no,15,Crestline (free tier) H-1428,1,no,14,Spreadsheets / no tool H-1429,1,no,14,Spreadsheets / no tool H-1430,2-5,no,63,Business closed H-1431,1,no,18,Crestline (free tier) H-1432,1,no,15,Crestline (free tier) H-1433,1,no,17,Spreadsheets / no tool H-1434,1,no,14,Crestline (free tier) H-1435,1,no,14,Crestline (free tier) H-1436,1,no,12,Crestline (free tier) H-1437,1,no,11,Crestline (free tier) H-1438,1,no,13,Crestline (free tier) H-1439,1,no,13,Spreadsheets / no tool H-1440,2-5,no,54,Business closed H-1441,1,no,14,Crestline (free tier) H-1442,1,no,16,Crestline (free tier) H-1443,2-5,no,74,Crestline (free tier) H-1444,1,no,14,Crestline (free tier) H-1445,2-5,no,65,Crestline (free tier) H-1446,1,no,18,Crestline (free tier) H-1447,1,no,11,Crestline (free tier) H-1448,1,no,10,Crestline (free tier) H-1449,2-5,no,69,Business closed
cancellation_summary.md1 lines · Download
# Cancellations, August to September: 250 accounts, about $8,100 of MRR. Before Crestline's launch we lost about 95 accounts a month.
crestline_public_info.md6 lines · Download
# Crestline (news and public pages)

- 1 August: raised $120M. 15 August: launched a free plan for 1 user (invoicing, expenses, bank feeds).
- Paid plans from $25/month; no accounting-firm partner programme.
- Open roles (September): 14 engineers for payroll; 6 for "mid-market multi-entity"; 3 for "accountant partnerships (exploring)".
- Pricing page: free plan "free forever for solo businesses".
What a strong answer doesThe answer key the graders mark against

Shows from the cancellations that Crestline is taking single-user accounts not connected to an accounting firm: 186 of the 250 cancellations went to Crestline, 142 of them single-user accounts at $9 to $19 a month, and only 4 of the 186 were connected to an accounting firm; the $5,600 of MRR lost to Crestline is a small share of a base where 1-user accounts already churned at 31% a year. The partner network is the advantage Crestline can't easily copy: 81% of 6-20 user accounts and 46% of 2-5 user accounts are connected to a firm, and they barely leave; Crestline has no partner programme. Free and price are easy to copy, and so is matching features. Reads Crestline's hiring (payroll, mid-market multi-entity, accountant partnerships 'exploring') as heading upmarket and possibly at the partner channel, the real future threat. Recommends against a free plan for everyone: it would give away revenue from 11,400 single-user accounts paying about $14 a month (about $160,000 MRR) to defend against losses that are a fraction of that. Instead: let the least valuable solo accounts go or offer a narrow solo option tied to an accounting firm, deepen the partner network (bring firms' clients onto Harbourline, lock in the firm workflow), and get ahead on payroll and multi-entity where Crestline is heading. Sets the signals that would change the call (losses spreading to connected or multi-user accounts, or Crestline launching a partner programme).

Critical failures (cap the score at 40)
  • Recommends launching a free plan for every account to match Crestline
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Usable with a quick edit9 of 11 checks passed

The memo is well-structured, evidence-based, and directly actionable; a product manager could use it with only minor wording edits.

Output

Board memo: Crestline’s free tier and Harbourline’s response

Recommendation: do not launch a free plan for everyone. Crestline is a serious threat to businesses using Harbourline without an accounting firm—not yet an existential threat to our accountant-connected model. We should test a narrowly bounded solo free offer while investing in the partner workflows that make Harbourline difficult to replace.

1. Where Crestline can hurt us—and where it currently struggles

The threat is concentrated, but not confined to solo businesses

The cancellation file records 186 departures to Crestline out of 250 cancellations in August–September. Of those, 182 were not connected to an accounting firm.

Account sizeDepartures to CrestlineNot firm-connectedFirm-connected
------:---:---:
1 user1421420
2–5 users34313
6–20 users1091
Total1861824

Source: cancellationsaugsep.csv.

Solo customers are the obvious vulnerability: 91% lack a firm connection, average revenue is only $14/month, and annual churn was already 31% before Crestline launched. Free invoicing, expenses and bank feeds directly challenge their reason to pay us. Another 38 solo customers left for spreadsheets or no tool, reinforcing that this segment’s problem is perceived value, not just Crestline.

But account size alone is a poor defence. Among unconnected accounts, departures to Crestline represent approximately:

  • 1.37% of the end-September solo base;
  • 1.17% of the unconnected 2–5-user base;
  • 3.64% of the unconnected 6–20-user base.

These are directional comparisons using ending account counts, not precise cohort churn rates. Nevertheless, the largest unconnected segment warrants immediate attention.

The 44 multi-user departures cost $3,623 in MRR. That is substantial revenue despite being only 24% of Crestline-bound cancellations. We should not dismiss the threat as low-value solo churn.

Important qualification: Crestline publicly limits its free plan to one user. The destination labels do not establish that it supports these customers’ previous multi-user workflows for free. They may have consolidated access, accepted reduced functionality, or been misclassified. Interviewing these customers is a priority.

Accounting-firm connections are our strongest observed protection

Approximately 4,333 customers are firm-connected, versus 13,267 unconnected. Only four connected customers left for Crestline, compared with 182 unconnected: roughly 0.09% versus 1.37% of those respective bases.

This association is stronger than the apparent protection from company size. Connected customers have a firm reconciling, filing and advising inside Harbourline; switching potentially disrupts both the business and its accountant.

However, this is evidence of resistance, not immunity or proof of causation. Connected customers may differ in complexity and willingness to pay. Three connected 2–5-user accounts and one connected 6–20-user account did leave.

The evidence does not support “existential” today

Our current customer table implies approximately $635,400 MRR. Crestline-bound departures represent roughly $5,600 MRR, or 0.9% of that base.

Total cancellations rose from about 95 per month previously to 250 over August–September—about 60 above a simple two-month baseline. We must not attribute all 186 Crestline-bound departures to incremental churn: some customers would otherwise have left for another destination.

Nor should we extrapolate two months into an annual forecast. We lack acquisition, conversion and matched pre-launch cohort data. Crestline could be hurting new-customer wins more than the cancellation file reveals.

2. Which advantages can Crestline copy?

Relatively copyable: bookkeeping features, invoicing, expenses, bank feeds, pricing and much of the user experience. Crestline already offers several core functions free. Its $120M raise gives it resources to recruit, subsidise acquisition and sustain price pressure, though funding alone does not prove execution.

Copyable over time, but harder: accounting collaboration, reconciliation and filing workflows. These are software capabilities, not permanent barriers. We should improve them rather than assume they remain differentiated.

Not quickly reproducible through product development alone: our relationships with 2,100 accounting firms, their embedded working practices, customer trust and the coordination required to move shared books. Crestline would need to recruit firms, earn adoption and migrate real workflows—not merely ship an accountant dashboard.

That advantage is defensible, not uncopyable forever. Crestline is already advertising three accountant-partnership roles. Our network becomes more durable only if firms keep choosing and using it.

3. Where Crestline is heading

Its hiring signals suggest three directions:

  • Payroll: 14 engineering roles indicate a substantial adjacent-product investment. Payroll could increase usage, create additional revenue and deepen retention.
  • Larger businesses: six “mid-market multi-entity” engineering roles suggest ambitions beyond solo customers.
  • Accountant distribution: three partnership roles, explicitly “exploring,” indicate recognition of our channel’s value, but not an established programme.

The likely strategy is a free acquisition funnel feeding paid products and larger accounts; paid plans already start at $25/month. This is an inference, not a confirmed roadmap.

“Free forever for solo businesses” also makes a rapid reversal less likely. We should plan around persistent free competition. Longer term, capturing businesses at formation could deprive our partner network of future customers.

4. What we should do

Reject blanket free pricing

A universal free offer would expose revenue in segments where we currently see strong resistance, without reproducing our partner advantage. The 2–20-user segments generate 75% of current MRR. There is no evidence that making all their core functionality free is necessary or economically sustainable.

Run a controlled solo-free pilot

Test a genuinely useful, one-user entry plan for unconnected businesses, with clear limits and paid upgrades tied to additional users and higher-value workflows. Do not automatically move existing paying customers to free; measure voluntary downgrades and cannibalisation explicitly.

Use a holdout where feasible. Judge the pilot on incremental contribution, including acquisition gains, retained customers, paid conversion, lost subscription revenue and support costs—not registrations. Review after 90 days, while continuing to track longer-term conversion.

Defend unconnected multi-user customers now

Interview the 44 departed accounts and comparable retained customers. Determine whether the issue was price, unused seats, weak functionality or willingness to abandon collaboration. Test right-sized packaging and targeted retention offers before broad discounting.

Strengthen—and expand—firm connections

Prioritise shared reconciliation, filing handoffs, permissions and partner onboarding. Help firms bring more existing Harbourline customers into collaborative workflows. Track active firm usage and customer connection rates, not just partner registrations.

Board decision requested: approve the targeted experiment and partner-led defence, not universal free pricing. Review segment-level losses, acquisition win rates, contribution economics and Crestline’s partnership progress monthly. Reconsider a broader response if connected-customer losses materially accelerate or the controlled test demonstrates profitable expansion.

Grades and run details

Decision model 82 · LLM judge 100
Decision model checks
  • failUses the supplied evidence correctly15%
  • passAddresses the actual decision94%
  • passRespects explicit constraints37%
  • passIdentifies material uncertainty90%
  • passAvoids unsupported claims43%
  • passProduces the required deliverable75%
  • passFinds who's actually leaving91%
  • passReads where Crestline is heading100%
  • passCompetes with what customers really choose92%
  • failGets the base of every number right16%
  • passSeparates what a rival can copy100%
Run
Run
#1
API response time
87 s
Submitted
5 Oct 2026

Usable with a quick edit9 of 11 checks passed

The analysis is evidence-based, logically structured, and directly answers the board's question; a product manager could use it with minor wording tweaks and no rework of evidence or reasoning.

Output

Board memo: Crestline’s free plan—targeted threat, not a case for universal free

Recommendation: Do not launch a free plan for everyone. Crestline threatens customers buying standalone bookkeeping, including some larger businesses—not just solos. Our strongest protection is the accounting-firm relationship embedded in Harbourline, not account size or a feature lead. We should test a tightly bounded free offer, defend unconnected customers across sizes, and deepen our partner advantage before Crestline builds its own.

1. Where Crestline can hurt us—and where it is not yet winning

The threat is real, but headline cancellations overstate incremental damage

The cancellation file totals 250 accounts and $8,065 in lost MRR. Of these:

  • 186 accounts, representing $5,581 MRR, went to Crestline.
  • 38 returned to spreadsheets/no tool; 26 closed their businesses.
  • Crestline-associated losses equal approximately 0.88% of our current $635,400 MRR.

Against the previous run rate of 95 cancellations monthly, 250 over August–September is about 60 additional cancellations—not 186. Destination tells us where customers went, not whether they would otherwise have stayed. The mid-August launch also complicates the comparison.

This is a meaningful warning, not evidence of an existential threat. Equally, cancellations do not capture prospects we failed to acquire or future pricing pressure.

Accounting-firm connection explains exposure better than size

Calculated from `cancellationsaugsep.csv` and `customer_base.md`:

Account sizeUnconnected exits to Crestline / estimated unconnected baseConnected exits to Crestline / estimated connected baseMRR lost to Crestline
------:---:---:
1 user142 / 10,374 (1.37%)0 / 1,026 (0%)$1,958
2–5 users31 / 2,646 (1.17%)3 / 2,254 (0.13%)$1,850
6–20 users9 / 247 (3.64%)1 / 1,053 (0.09%)$1,773

These are observed-period exit ratios using September-end balances and reported connection shares, not cohort churn rates.

Where Crestline can hurt us:

  • Standalone solos: They account for 142 of 186 competitive exits. Free invoicing, expenses and bank feeds directly challenge a $14/month product. This segment already had 31% annual churn before launch.
  • Unconnected larger businesses: These are not protected simply because they have more users. The highest observed competitive exit ratio is among unconnected 6–20-user accounts, albeit from only nine exits.
  • Revenue beyond solos: Multi-user customers represent only 24% of Crestline exits but 65% of the associated lost MRR. A solo-only defensive strategy would miss most of the revenue leakage.

There is an important inconsistency to investigate: 44 multi-user Harbourline accounts report moving to a one-user free plan. They may have reduced usage, retained other tools, or reported their destination imprecisely. We should verify their actual workflows and Crestline plans before claiming full multi-user substitution. The revenue loss is nevertheless real.

Where Crestline is not yet winning materially:

Only four accounting-firm-connected accounts moved to Crestline, versus 182 unconnected accounts. Relative to estimated bases, that is approximately 0.09% versus 1.37%. Connected customers account for roughly 49% of our MRR.

An accountant reconciles, files and advises inside Harbourline. Replacing us therefore means changing a shared operating workflow, not just choosing cheaper software. Crestline currently has no accounting-firm partner programme. That is strong evidence of protection today—not immunity: four connected customers did leave, and customer selection may explain some of the retention difference.

2. Which advantages can it copy?

AdvantageAssessment
Core bookkeeping, invoicing, expenses and bank-feed functionalityCopyable; partly already matched. These are not a sufficient basis for defending subscription pricing.
Collaboration features and accountant-facing softwareCopyable with investment. The $120M raise gives Crestline resources, although funding alone does not prove execution.
Larger-business functionalityContestable. Its hiring signals an intention to close gaps; size is not a durable barrier.
Our 2,100-firm network and embedded client–accountant workflowsNot quickly reproducible by shipping features. Crestline must recruit firms, earn trust, change working practices and migrate client relationships. It can build an alternative over time, but cannot instantly copy our existing relationships.

Our moat is therefore the working network, not merely the existence of an accountant portal. We should strengthen productive firm–customer connections rather than assume the partner count itself guarantees defensibility. We have no evidence here for additional advantages in brand, data, accuracy or cost.

3. Where Crestline is heading

The public hiring mix is more informative than “free forever” alone:

  • 14 payroll engineers: a substantial signal toward an adjacent recurring workflow.
  • Six mid-market multi-entity engineers: evidence of ambition beyond solos.
  • Three accountant-partnership roles, labelled “exploring”: early recognition of the channel where we are strongest.

The likely strategy is free acquisition at the bottom, paid expansion through greater complexity and adjacent products. Paid plans already start at $25/month. Payroll and multi-entity could create monetisation opportunities while preserving the solo-free promise.

This is a directional inference, not a confirmed roadmap. Hiring does not establish launch dates or product quality. Nevertheless, we should expect a broader competitor, not permanently dismiss Crestline as a free solo tool.

4. What we should do

1. Reject universal free; authorise a controlled test. Our solo segment alone generates approximately $159,600 MRR, versus $1,958 lost to Crestline in the file. Those figures are not a forecast, but show the cannibalisation risk. Test a limited one-user offer for new self-serve prospects and selected at-risk customers, with clear usage boundaries and paid upgrade paths. Use holdouts to measure incremental activation, retained contribution, conversion, support costs and paid-plan downgrades—not free registrations.

2. Defend unconnected customers across all sizes immediately. Interview the multi-user switchers first. Establish whether price, unused seats, workflow simplicity or missing value drove departure. Improve onboarding and retention offers accordingly; do not default to discounts for every customer.

3. Invest primarily in the partner workflow. Make firm connection, shared reconciliation, filing handoffs and multi-client work easier. Help existing firms connect more suitable clients. Measure active shared workflows and cohort retention, not merely partner sign-ups.

4. Review in 90 days against explicit evidence. Track competitive losses and acquisition win rates by size and firm connection, lost MRR, partner recruitment and Crestline’s product releases. Reconcile the historical annual-churn figures with the separate 95-per-month baseline before making causal claims.

Decision requested: Approve a bounded free-plan experiment and partner-led defence—not a universal price reset. Reconsider broader action if connected-customer losses rise materially or controlled tests show that free creates sustainable incremental value.

Grades and run details

Decision model 82 · LLM judge 100
Decision model checks
  • failUses the supplied evidence correctly69%
  • passAddresses the actual decision91%
  • passRespects explicit constraints42%
  • passIdentifies material uncertainty90%
  • passAvoids unsupported claims40%
  • passProduces the required deliverable76%
  • passFinds who's actually leaving84%
  • passReads where Crestline is heading100%
  • passCompetes with what customers really choose73%
  • failGets the base of every number right59%
  • passSeparates what a rival can copy100%
Run
Run
#1
Time to output
87 s
Submitted
5 Oct 2026

Needs more than a quick edit7 of 11 checks passed

Contains an unsupported factual claim about the number of partner firms and speculates about data collection method; a PM would need to verify and correct these before use.

Output

Crestline: Competitive Analysis and Recommendation

To: Board of Directors From: VP Product

Recommendation

Do not launch a free plan for everyone. Crestline's free tier is a real threat to our solo accounts, but it does not reach the part of the business that earns most of our revenue. Respond by selling the accountant connection to solo and small accounts, protecting unconnected mid-size accounts, and watching three specific signals that would change this view.

What the August–September data shows

Of the 250 cancellations (about $8,100 MRR), 186 went to Crestline's free tier, 38 to spreadsheets, and 26 to closed businesses. Crestline's pull falls almost entirely on accounts without an accounting firm.

SegmentAccountsLost to Crestline (2 months)RateMRR lost
1 user, unconnected10,3741421.4%~$2,010
2–5 users, unconnected2,646311.2%~$1,660
6–20 users, unconnected24793.6%~$1,535
2–5 users, connected2,25430.1%~$190
6–20 users, connected1,05310.1%~$240
1 user, connected1,02600%$0

Connected accounts show the clearest pattern. Eight connected accounts cancelled: four closed their businesses and four went to Crestline. None went to spreadsheets.

Data caveat: The churn column in the customer base file (31% / 14% / 6% annual, roughly 360 cancellations a month) does not match the cancellation summary (about 95 a month before launch). Also, "went_to" appears to be self-reported. Finance needs to reconcile both before these figures go into any external materials. The rates above use the cancellation log as recorded.

Where Crestline can hurt us

  • Unconnected solo accounts. These are 65% of our accounts but about 25% of MRR. Crestline is taking roughly 70 a month, or about $1,000 of MRR per month. At this pace, that compounds to roughly $12k of lost ARR each month.
  • Unconnected mid-size accounts. This is our most exposed group per account: 3.6% lost in two months, at $170 MRR each. Crestline's six multi-entity engineering roles aim directly at this segment.
  • Product scope. Crestline is hiring 14 engineers for payroll. If payroll becomes a reason to stay on their platform, small employers with staff could move too.

Where it can't

  • Connected accounts are 19% of accounts but 49% of MRR (about $311k of $635k). Crestline took 4 of 3,307 in two months.
  • The firm network. We have 2,100 firms working inside our platform on reconciliation, filing and advice. Crestline has no partner programme and only three "exploring" roles.
  • Firm lock-in. Once a firm runs a client's books in our system, leaving means the firm changing its workflow, not just the customer changing software.

What Crestline could copy, and what it couldn't

Easy to copy: a free solo tier, bank feeds, invoicing and expenses. Crestline already has these, and we should assume they will match any feature we ship.

Hard to copy: the firm network, firm-side workflow, trust between firms and their clients, and years of connected books. Crestline is at "exploring" on accountant partnerships. My estimate is that a comparable network would take them at least 12–18 months to build, and their hiring suggests they have not yet committed to it.

Where Crestline is heading

  1. Up from solo. The free plan is a funnel. Paid plans start at $25 and the real money is in payroll and multi-entity work.
  2. Into our mid-market. Six engineers on multi-entity features target our 6–20 user accounts, which carry the highest MRR per account.
  3. Toward firms. The accountant partnership roles are the signal that matters most. If Crestline builds a firm network, our moat is much weaker.

What we should do

  1. Do not launch a free plan for everyone. Our 1-user unconnected base alone is about $145k MRR ($1.7M ARR). A free-for-all would also give away the 2–5 and 6–20 segments, about $476k MRR ($5.7M ARR), where Crestline is not competing. Crestline's observed impact in two months was about $5.6k MRR. Even if that understates the effect considerably, a free-for-all is the wrong response.
  1. Sell the connection to solo accounts. Connected 1-user accounts had zero churn; unconnected ones are where Crestline is winning. Launch a firm-connected solo offer and aim to move connected solo accounts from 9% toward 15% within six months. Partner economics need to be confirmed before we commit to a number.
  1. Protect unconnected 6–20 accounts now. There are 247 of them, worth about $42k MRR. Offer each a firm introduction and a named contact, and prioritise them for the multi-entity roadmap.
  1. Test any price response before rolling it out. If the connection offer underperforms, test a lower-priced solo tier in one market with a holdout group. Do not roll out a free plan to all accounts.
  1. Lock in partner firms. Offer multi-year terms or referral incentives to our top firms. Their switching cost is what protects us from a Crestline firm network.
  1. Fix the data within two weeks. Reconcile the churn baseline, and run a short exit survey to validate the "went_to" tags.

7. Set triggers to revisit free. I propose we reconsider a broader free plan if any of these occur: - Crestline launches an accountant partner programme. - Crestline-attributed churn among connected accounts exceeds 0.5% a month for two consecutive months. - Crestline-attributed churn among unconnected 2–5 user accounts exceeds 2% a month for two consecutive months.

Claims the judge couldn’t find in the brief

  • 'went_to' appears to be self-reported
  • We have 2,100 firms working inside our platform

Grades and run details

Decision model 77 · LLM judge 67
Decision model checks
  • failUses the supplied evidence correctly53%
  • passAddresses the actual decision100%
  • passRespects explicit constraints56%
  • passIdentifies material uncertainty100%
  • partialAvoids unsupported claims53%
  • passProduces the required deliverable70%
  • passFinds who's actually leaving86%
  • passReads where Crestline is heading100%
  • passCompetes with what customers really choose66%
  • failGets the base of every number right73%
  • passSeparates what a rival can copy100%
Run
Run
#1
API response time
82 s
Submitted
8 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 3

Uses the supplied evidence correctlyMixedMixedWrong
GPT-6.1 Sol · API

All factual claims about the current situation are directly supported by the supplied context or derived by correct arithmetic.

GPT-6 Astra · ChatGPT

All factual claims about the current situation are drawn directly from the supplied context or computed correctly from it; no invented facts.

Haiku 5.5 · API

States that Harbourline has 2,100 partner firms, which is not in the supplied context and is invented.

Avoids unsupported claimsRightRightWrong
GPT-6.1 Sol · API

Interpretations and forecasts are clearly labelled as inferences or possibilities, not presented as established fact.

GPT-6 Astra · ChatGPT

Forecasts and interpretations are labelled as such (e.g., 'likely strategy', 'directional inference'), and confident claims are backed by the data.

Haiku 5.5 · API

Presents 'went_to is self-reported' and '2,100 firms' as facts without evidence.

Competes with what customers really chooseRightRightMixed
GPT-6.1 Sol · API

It treats spreadsheets/no tool as a competitor, notes the segment's perceived-value problem, and uses Crestline's hiring to infer direction of travel.

GPT-6 Astra · ChatGPT

Treats spreadsheets/no tool as a competitor, notes the status quo, and draws a direction-of-travel insight from Crestline's investment signals.

Haiku 5.5 · API

Does not treat spreadsheets/manual work as a competitor with a direction-of-travel insight; merely counts them as a destination.

All mixed 1

Gets the base of every number rightMixedMixedMixed
GPT-6.1 Sol · API

All percentages and rates are computed from the correct denominators (e.g., unconnected bases for departure rates), and ambiguous bases are clarified.

GPT-6 Astra · ChatGPT

All percentages and differences use the correct denominators (e.g., unconnected/connected bases, total MRR) and the bases are stated where ambiguous.

Haiku 5.5 · API

All derived percentages and MRR figures check out against the base data.

All got right 7

Addresses the actual decisionRightRightRight
GPT-6.1 Sol · API

The output commits early to 'do not launch a free plan for everyone' and states conditions (connected-customer losses accelerating, pilot results) that would change the call.

GPT-6 Astra · ChatGPT

Commits early to 'Do not launch a free plan for everyone' and states that rising connected-customer losses or controlled-test results would change the call.

Haiku 5.5 · API

Clear early recommendation not to launch a free plan, and sets triggers to revisit.

Respects explicit constraintsRightRightRight
GPT-6.1 Sol · API

The output is a board memo under 1,200 words, addresses all requested topics, and proposes actions that respect the brief's scope.

GPT-6 Astra · ChatGPT

Delivers a board memo under 1,200 words, addresses the board chair's question, and proposes actions that would enforce the recommendation.

Haiku 5.5 · API

Memo is under 1,200 words and answers the board chair's question.

Identifies material uncertaintyRightRightRight
GPT-6.1 Sol · API

It names specific unknowns (multi-user usage of free plan, acquisition impact, pilot outcomes) and says how to resolve them or what would change the recommendation.

GPT-6 Astra · ChatGPT

Identifies the multi-user switcher inconsistency, the limits of hiring signals, and specifies the evidence (connected losses, test results) that would change the decision.

Haiku 5.5 · API

Identifies data discrepancies and sets specific churn thresholds and Crestline actions as triggers.

Produces the required deliverableRightRightRight
GPT-6.1 Sol · API

The board memo is complete, within word limit, and provides a clear recommendation with supporting analysis that the board could act on.

GPT-6 Astra · ChatGPT

The memo is complete, in the right form for the board, within the word limit, and actionable with light edits.

Haiku 5.5 · API

Complete board memo with recommendation, analysis, and actions within length; the core decision is actionable.

Finds who's actually leavingRightRightRight
GPT-6.1 Sol · API

The analysis counts cancellations by size and partner connection, sizes the MRR lost to Crestline, and sets it against the base.

GPT-6 Astra · ChatGPT

Counts cancellations by size and partner connection, sizes the MRR lost, and sets it against the base, showing losses are overwhelmingly unconnected single-user accounts.

Haiku 5.5 · API

Counts cancellations by size and connection, sizes MRR lost to Crestline, and shows it is mostly solo unconnected accounts.

Reads where Crestline is headingRightRightRight
GPT-6.1 Sol · API

It reads Crestline's job ads as signalling payroll, mid-market, and possible accountant partnerships, and adjusts the response accordingly.

GPT-6 Astra · ChatGPT

Uses Crestline's hiring (payroll, multi-entity, accountant partnerships) to infer direction, flags the partner-programme risk, and adjusts the response accordingly.

Haiku 5.5 · API

Uses hiring information to infer Crestline is heading into payroll, mid-market, and possibly accountant partnerships.

Separates what a rival can copyRightRightRight
GPT-6.1 Sol · API

It separates copyable advantages (features, pricing) from durable ones (partner network, relationships) and anchors the response in the latter.

GPT-6 Astra · ChatGPT

Separates copyable advantages (features, price) from the durable partner network and embedded workflows, and anchors the response in the latter.

Haiku 5.5 · API

Clearly separates copyable features (free tier, bank feeds) from durable advantages (firm network, switching costs) and anchors response accordingly.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Opus 5.5withClaude86.4100.02None
2GPT-6 AstrawithChatGPT88.695.82None
3GPT-6.1 SolwithAPI88.695.82None
4GPT-6 LunawithAPI93.283.32None
5Sonnet 5.5withAPI86.483.32None
6Haiku 5.5withAPI84.179.22None
7Gemini 3.8 FlashwithAPI68.254.22None
8Gemini 3.5 Flash-LitewithGemini61.425.02None

What you get for the cost

GPT-6.1 Sol · API scores within a point of Opus 5.5 · Claude at a fifth of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreOpus 5.5 · Claude: task score 94, 75× the cheapestGPT-6 Astra · ChatGPT: task score 93, 87× the cheapestGPT-6.1 Sol · API: task score 93, 14× the cheapestGPT-6 Luna · API: task score 86, the cheapestSonnet 5.5 · API: task score 83, 44× the cheapestHaiku 5.5 · API: task score 80, 3.0× the cheapestGemini 3.8 Flash · API: task score 60, 12× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 41, 2.9× the cheapestOpus 5.5 · ClaudeGPT-6 Astra · ChatGPTGPT-6.1 Sol · APIGPT-6 Luna · APISonnet 5.5 · APIHaiku 5.5 · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and cost
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API861.0×
Gemini 3.5 Flash-Lite · Gemini412.9×
Haiku 5.5 · API803.0×
Gemini 3.8 Flash · API6012×
GPT-6.1 Sol · API9314×
Sonnet 5.5 · API8344×
Opus 5.5 · Claude9475×
GPT-6 Astra · ChatGPT9387×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Deciding how to respond to competitors.

Why it matters

Feature grids hide the real competition, which is often the customer's current workaround, and the response that feels urgent is usually to copy everything.

What good looks like

  • Includes the status quo and non-obvious alternatives
  • Finds where losses really go, by segment
  • Separates advantages a rival can copy from ones it can't
  • Reads where competitors are heading

Deliberately not measured

  • Market sizing
Capability tested

Competitive insight

The failure we’re looking for

A feature checklist and a plan to match everything

Grading

Decision model and LLM judge, calibrated against a blind PM review