Humanity’s Last PRD
Latest drop: Haiku 5.5 · 8 Oct
Tasks / Strategy

Competitor analysis

Can the model find where a competitor can actually hurt you, from win/loss and churn data, rather than list features?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 63% were usable with at most a quick edit.

Reliably right

  1. Respects explicit constraints100% pass
    Delivers a competitive analysis memo to Dan within 900 words, addressing all requested topics without proposing a full feature-by-feature match.
    Opus 5.5 · Claude · The competitor isn't who you think
  2. Reads where Crestline is heading100% pass
    Uses Crestline's hiring (payroll, multi-entity, accountant partnerships) to infer direction, flags the partner-programme risk, and adjusts the response accordingly.
    GPT-6 Astra · ChatGPT · Free forever, funded
  3. Produces the required deliverable97% pass
    The memo is complete, in the right form for the CEO, within the word limit, and actionable with light edits.
    Opus 5.5 · Claude · The competitor isn't who you think

Where it slips

  1. Uses the supplied evidence correctly56% pass
    Two factual claims are wrong: the number of unconnected multi-user accounts going to Crestline is 40, not 42, and the number of connected 6-20 cancellations is 5, not 4.
    Sonnet 5.5 · API · Free forever, funded
  2. Gets the base of every number right61% pass
    Derived figures for unconnected multi-user cancellations (42 vs actual 40) and connected 6-20 cancellations (4 vs actual 5) are incorrect.
    Sonnet 5.5 · API · Free forever, funded
  3. Separates what a rival can copy66% pass
    Does not explicitly separate advantages a well-funded competitor could copy from those it couldn't, nor anchor the response in durable advantages with evidence.
    GPT-6 Luna · API · The competitor isn't who you think

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Competes with what customers really choose

    Does the analysis treat the status quo (spreadsheets, manual work, doing nothing) as a competitor, and draw at least one implication from where competitors are investing rather than only from today's features?

    Passes when Analyses the status quo, strengths included, and gives a direction-of-travel insight that affects the recommendation.

    April DunfordLenny’s Podcast

  2. Separates what a rival can copy

    Does the analysis separate advantages a well-funded competitor could copy (features, price, being first) from ones it couldn't easily (switching costs, networks, relationships), and build the response on the latter?

    Passes when Names which advantages are copyable and which aren't, with evidence, and anchors the response in the durable ones.

    Hamilton HelmerLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for growth at Tallybook. Our CEO, Dan Whitlock, wants a feature-by-feature comparison against Billfront and a plan to match everything they have by Q2 (his note is below). Write him a competitive analysis of no more than 900 words: who we really lose to, where Billfront is a real threat and where it isn't, where it's heading, and what we should do instead of (or as well as) matching it. The files are attached.

What the model was given5 items: About Tallybook, ceo_note.md, deals_last_two_quarters.csv (trials that became deals, won or lost), customers_by_segment.md, billfront_public_info.md
About TallybookInvoicing and payments for freelancers and small businesses. Plans from $9 to $39 a month.
ceo_note.md1 lines · Download
From Dan (CEO): "Billfront is killing us. I want a full feature-by-feature comparison against them and a plan to match everything they have by Q2."
deals_last_two_quarters.csv (trials that became deals, won or lost)deal,segment,outcome,primary_reason,went_to T-900,Small business,won,, T-901,Freelancer,won,, T-902,Freelancer,won,, T-903,Small business,won,, T-904,Small business,lost,Accountant uses something else,Xero invoicing T-905,Small business,won,, T-906,Small business,lost,Didn't get round to switching,No decision T-907,Freelancer,won,, T-908,Freelancer,won,, T-909,Small business,won,, T-910,Freelancer,lost,Price,Billfront T-911,Freelancer,won,, T-912,Small business,lost,Price,Billfront T-913,Freelancer,won,, T-914,Freelancer,won,, T-915,Small business,won,, T-916,Freelancer,won,, T-917,Small business,won,, T-918,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-919,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-920,Freelancer,won,, T-921,Agency (multi-currency),won,, T-922,Small business,won,, T-923,Small business,won,, T-924,Small business,won,, T-925,Freelancer,won,, T-926,Freelancer,won,, T-927,Freelancer,won,, T-928,Small business,won,, T-929,Small business,lost,Didn't get round to switching,No decision T-930,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-931,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-932,Small business,won,, T-933,Small business,won,, T-934,Small business,won,, T-935,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-936,Agency (multi-currency),lost,Didn't get round to switching,No decision T-937,Small business,lost,Didn't get round to switching,No decision T-938,Agency (multi-currency),won,, T-939,Small business,lost,Didn't get round to switching,No decision T-940,Small business,won,, T-941,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-942,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-943,Freelancer,won,, T-944,Small business,won,, T-945,Small business,lost,Accountant uses something else,Xero invoicing T-946,Small business,won,, T-947,Small business,won,, T-948,Small business,won,, T-949,Small business,lost,Didn't get round to switching,No decision T-950,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-951,Freelancer,won,, T-952,Freelancer,won,, T-953,Freelancer,won,, T-954,Agency (multi-currency),won,, T-955,Small business,lost,Didn't get round to switching,No decision T-956,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-957,Freelancer,won,, T-958,Small business,won,, T-959,Freelancer,won,, T-960,Small business,lost,Didn't get round to switching,No decision T-961,Freelancer,won,, T-962,Small business,won,, T-963,Agency (multi-currency),won,, T-964,Small business,won,, T-965,Small business,won,, T-966,Small business,won,, T-967,Small business,lost,Didn't get round to switching,No decision T-968,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-969,Small business,won,, T-970,Freelancer,lost,Price,Billfront T-971,Small business,lost,Didn't get round to switching,No decision T-972,Small business,lost,Accountant uses something else,Xero invoicing T-973,Small business,lost,Accountant uses something else,Xero invoicing T-974,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-975,Small business,lost,Price,Billfront T-976,Freelancer,won,, T-977,Small business,lost,Didn't get round to switching,No decision T-978,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-979,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-980,Freelancer,won,, T-981,Small business,won,, T-982,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-983,Small business,won,, T-984,Small business,lost,Didn't get round to switching,No decision T-985,Freelancer,won,, T-986,Freelancer,won,, T-987,Small business,lost,Didn't get round to switching,No decision T-988,Small business,won,, T-989,Freelancer,won,, T-990,Freelancer,won,, T-991,Freelancer,won,, T-992,Agency (multi-currency),won,, T-993,Freelancer,won,, T-994,Freelancer,won,, T-995,Small business,won,, T-996,Small business,won,, T-997,Small business,won,, T-998,Small business,lost,Didn't get round to switching,No decision T-999,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-1000,Agency (multi-currency),won,, T-1001,Small business,lost,Price,Billfront T-1002,Small business,won,, T-1003,Freelancer,won,, T-1004,Agency (multi-currency),lost,Didn't get round to switching,No decision T-1005,Freelancer,won,, T-1006,Small business,won,, T-1007,Freelancer,won,, T-1008,Small business,won,, T-1009,Freelancer,lost,Price,Billfront T-1010,Small business,won,, T-1011,Freelancer,won,, T-1012,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-1013,Freelancer,won,, T-1014,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-1015,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-1016,Freelancer,won,, T-1017,Freelancer,won,, T-1018,Freelancer,won,, T-1019,Small business,lost,Didn't get round to switching,No decision T-1020,Freelancer,won,, T-1021,Small business,won,, T-1022,Agency (multi-currency),won,, T-1023,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-1024,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-1025,Freelancer,won,, T-1026,Freelancer,won,, T-1027,Freelancer,won,, T-1028,Small business,won,, T-1029,Freelancer,won,,
customers_by_segment.md7 lines · Download
# Tallybook customers and revenue (September)

| Segment | Paying customers | Average revenue per month |
|---|---|---|
| Freelancer | 5,800 | $11 |
| Small business | 3,100 | $24 |
| Agency (multi-currency) | 240 | $46 |
billfront_public_info.md6 lines · Download
# Billfront (public pages, September)

- Pricing: $12/month Starter, $29/month Pro (multi-currency, live FX rates), $59/month Team.
- Homepage: "Invoicing for global teams". Case studies: three agencies, one consultancy.
- Recent launches: multi-currency invoicing (March), client portal (June).
- Job posts: 4 engineers for "global payments", 2 for "agency workflows".
What a strong answer doesThe answer key the graders mark against

Shows from the deals file that the biggest competitor is no decision: of 46 lost deals, 25 ended with the prospect staying on spreadsheets or never switching, 17 went to Billfront and 4 to Xero invoicing. Billfront's wins are concentrated in multi-currency agencies (11 of its 17), on a specific gap (invoicing in several currencies with live rates); elsewhere it won only 6 deals, mostly on price. Agencies are a small segment (240 customers at about $46 a month, about $11,000 MRR, against about $64,000 from freelancers and $74,000 from small businesses), and Billfront is heading further into them (global payments and agency workflow hiring). Recommends against matching everything: fix switching friction for small businesses who 'didn't get round to it' (imports, setup help, accountant tie-ins), make a deliberate call on multi-currency (build it if agencies matter strategically, or cede the segment), and keep the advantages Billfront can't easily copy. Gives a short comparison only where it matters.

Critical failures (cap the score at 40)
  • Recommends matching Billfront feature for feature
Task

Mid-level · v1.0

The outputs

8 setups answered this brief.

Opus 5.5Claude98Usable with a quick editGPT-6 LunaAPI94Usable with a quick editGPT-6 AstraChatGPT94Usable with a quick editSonnet 5.5API94Usable with a quick editGPT-6.1 SolAPI94Usable with a quick editHaiku 5.5API92Usable with a quick editGemini 3.8 FlashAPI71Usable with a quick editGemini 3.5 Flash-LiteGemini35Needs more work
GPT-6.1 SolAPI

Usable with a quick edit10 of 11 checks passed

The analysis is evidence-based, actionable, and requires no re-checking of facts or reasoning; a PM could use it with light edits.

Output

Competitive analysis: Billfront is a targeted threat—not our biggest source of losses

To: Dan Whitlock From: Growth PM

Recommendation: Do not commit to matching every Billfront feature by Q2. Commit to reducing switching friction in our core market, testing price objections, and validating a focused multi-currency response. Maintain a competitive feature inventory, but let customer evidence—not competitor launches—drive delivery.

1. Who we actually lose to

Across the 130 trial opportunities in the last two quarters, we won 84 and lost 46:

Loss destinationDealsShare of lossesRecorded reason
------:---:---
No decision2554%16 never switched; 9 stayed with spreadsheets/Word
Billfront1737%11 needed multi-currency/live rates; 6 cited price
Xero invoicing49%Accountant preference

Our largest competitor is inertia. More opportunities stayed with their existing process than chose Billfront. Feature parity does not directly address those losses.

Billfront’s wins are concentrated: 11 of its 17 wins came from multi-currency agencies. Among the 110 freelancer and small-business opportunities, only six went to Billfront—all citing price.

These are trial-opportunity outcomes, not evidence of customer churn or overall market share. We also lack deal values, so counts alone cannot establish the revenue impact.

2. Where Billfront is—and is not—a real threat

Real threat: international agencies. Of 20 agency opportunities, we won seven, lost 11 to Billfront over multi-currency/live rates, and lost two to no decision. That is a specific, repeated capability gap, not a vague brand problem.

Agencies are a small but valuable segment: 240 paying customers generate approximately $11,040 monthly revenue, or 7.4% of the total, at the highest segment average of $46 per month. Protecting this segment merits investigation; pursuing it aggressively requires a deliberate market choice.

Limited demonstrated threat: our core market. Freelancers and small businesses represent 97.4% of paying customers and 92.6% of monthly revenue. Their recorded Billfront losses concern price, not missing portals or team workflows.

That does not mean price is objectively lower: Tallybook starts at $9 versus Billfront’s $12. We need to understand required tiers, perceived value, discounts, and payment costs before changing prices.

Separate threat: accountant-led selection. Four small businesses chose Xero because their accountant used it. Matching Billfront does nothing to resolve that workflow preference.

3. Feature comparison: what the evidence supports

A full parity assessment is not possible from these files: we lack Tallybook’s feature inventory and detailed Billfront documentation.

AreaBillfront evidenceTallybook evidence / implication
InvoicingPublic positioning and launches confirm offeringCore Tallybook offering; workflow-level parity unverified
Payments“Global payments” hiringCore Tallybook offering; global capabilities unverified
Pricing$12 Starter; $29 Pro; $59 Team$9–$39 plans; six price losses require tier-level investigation
Multi-currency + live FXIncluded in $29 Pro; launched MarchEleven losses establish an unmet requirement; exact existing coverage needs audit
Client portalLaunched JuneAvailability unknown; no recorded loss cites it
Team/agency workflowsTeam plan; agency-workflow hiringDetailed features unknown; no evidenced parity requirement yet

We should complete this inventory through product testing and documentation—not turn unknowns into assumed deficiencies.

4. Where Billfront appears to be heading

“Invoicing for global teams,” three agency case studies, one consultancy case study, and hiring for global payments and agency workflows point toward cross-border agency operations, rather than primarily low-cost freelancer invoicing.

The multi-currency launch and portal fit that direction. Hiring suggests possible investment in global payment capabilities and deeper agency workflows, but does not confirm particular upcoming features.

Following that trajectory wholesale would move us toward a different customer and potentially greater operational complexity.

5. What we should deliver by Q2

First: establish the facts. Product and Growth should audit both products, interview the 17 Billfront losses and a sample of no-decision opportunities, and identify the exact currency, payment, migration, and price requirements. Validate engineering scope before promising parity.

Prioritize core conversion. Test guided setup, invoice/customer import, template migration, and assisted switching. Measure first-invoice activation, time to first invoice, and trial-to-paid conversion against a control—not just onboarding completion.

Address price without a blanket cut. Test clearer tier/value messaging and targeted offers for price-sensitive prospects. Judge success on paid conversion, retained revenue, and margin.

Gate a focused FX investment. Scope multi-currency invoicing with live rates—not the entire global-payments stack. Proceed only if recoverable agency revenue, customer demand, and delivery cost justify it. Use agency design partners and explicit pilot adoption and retention criteria.

Investigate accountant compatibility. Determine whether exports, accountant access, or integrations would resolve the Xero losses before building.

Q2 commitment: validated competitive coverage, measured conversion experiments, and an evidence-based FX build decision—not indiscriminate feature parity.

Grades and run details

Decision model 95 · LLM judge 92
Decision model checks
  • passUses the supplied evidence correctly46%
  • passAddresses the actual decision89%
  • passRespects explicit constraints36%
  • passIdentifies material uncertainty100%
  • passAvoids unsupported claims64%
  • passProduces the required deliverable51%
  • passFinds the real competitor100%
  • passPins down where Billfront wins100%
  • passCompetes with what customers really choose100%
  • passGets the base of every number right44%
  • partialSeparates what a rival can copy61%
Run
Run
#1
API response time
48 s
Submitted
5 Oct 2026
Haiku 5.5API

Usable with a quick edit10 of 11 checks passed

The output provides a data-backed, actionable memo. The omission of a copyable vs. durable advantage distinction is a small gap that could be fixed with a quick edit without rechecking evidence or redoing reasoning.

Output

To: Dan Whitlock From: PM, Growth Re: Billfront competitive analysis

Bottom line: Billfront is beating us in one segment: multi-currency agencies. Elsewhere we mostly lose to inertia, not to them. Matching Billfront feature by feature would spend Q2 on the wrong problem. I recommend a scoped multi-currency build for agencies plus a push to convert stalled trials.

Who we actually lose to

Over the last two quarters we won 84 of 130 trials (65%) and lost 46:

Loss reasonLossesShare
No decision: "didn't get round to switching"1635%
No decision: stayed on spreadsheets / Word920%
Billfront: multi-currency with live rates1124%
Billfront: price613%
Accountant uses Xero49%

Fifty-four percent of our losses go to doing nothing. Billfront accounts for 37%.

Segment matters more than count. Agencies win only 7 of 20 trials (35%), against 77% for freelancers and 63% for small businesses. Eleven of our 13 agency losses went to Billfront. Agencies are 240 customers paying $46 a month, so weighting lost deals by segment revenue, Billfront-attributed losses come to roughly $610 a month, and about 83% of that is multi-currency agencies.

Where Billfront is a real threat

  • Multi-currency agencies. Their $29 Pro tier sells live FX rates, and we don't match that for this buyer. This is the one feature gap costing us repeatedly.
  • Their direction. The homepage says "Invoicing for global teams," three of their four case studies are agencies, and they launched multi-currency in March. Their job posts show four engineers on global payments and two on agency workflows. They're moving toward the segment where we're weakest.

Where it isn't

  • Freelancers and small businesses. Only 6 of our 46 losses went to Billfront on price, and none cite a missing feature. Most losses in these segments are status quo.
  • Price. Billfront's Starter is $12 against our $9 entry, so their cheapest plan costs more than ours. The price losses are probably at our mid tier against their Pro or Starter. I'd need the deal notes to confirm, but six deals don't justify a broad price cut.
  • Client portal. Billfront launched it in June, and no lost deal mentions it.

What to do instead of, or as well as, matching

  1. Scope live-FX multi-currency invoicing for agencies (Q2 target). Build what agencies actually need, not Billfront's full feature list. Integrating an FX rate feed will likely be faster than building rate infrastructure. Target: lift agency win rate from 35% to 50% or more.
  2. Work the 25 no-decision losses. "Didn't get round to switching" and "stayed on spreadsheets" point to trial follow-up and migration problems. Test a spreadsheet and Word-template importer, a personal check-in from sales on high-value trials around day 7, and a time-limited switch offer. Run these in two-week cycles and measure conversion.
  3. Address the accountant problem (4 losses). Let accountants view and collaborate on invoices, and check our Xero sync coverage.
  4. Don't chase parity. Skip the client portal and other Billfront features unless losses start citing them. Track loss reasons monthly so future roadmap decisions rest on data.

Caveats

  • 130 trials is a small sample, and loss reasons are rep-recorded. The agency figures rest on just 20 deals.
  • Agency ARPU of $46 is above our $39 top plan, so these customers are probably on add-ons or custom pricing. I'll confirm this before we size the multi-currency opportunity.
  • Billfront's pricing and job posts are public signals, not a confirmed roadmap.

Ask: Approve a two-week scoping sprint for agency multi-currency and a six-week trial-conversion test. I'll bring back scope, cost, and expected impact by the end of the sprint.

Grades and run details

Decision model 91 · LLM judge 92
Decision model checks
  • passUses the supplied evidence correctly25%
  • passAddresses the actual decision62%
  • passRespects explicit constraints67%
  • passIdentifies material uncertainty89%
  • partialAvoids unsupported claims27%
  • passProduces the required deliverable69%
  • passFinds the real competitor97%
  • passPins down where Billfront wins86%
  • passCompetes with what customers really choose100%
  • passGets the base of every number right9%
  • partialSeparates what a rival can copy73%
Run
Run
#1
API response time
32 s
Submitted
8 Oct 2026
Sonnet 5.5API

Usable with a quick edit10 of 11 checks passed

The analysis is clear, data-backed, and actionable; a PM could use it with minor wording edits without rechecking evidence or redoing reasoning.

Output

To: Dan Whitlock From: Growth PM Re: Billfront: what the deal data says

Bottom line

Billfront is a real threat in one segment, not across the board. Most of our losses aren't to Billfront at all. I recommend we don't match everything by Q2. We should build the one feature that is costing us deals, and fix the larger problem, which is trials that never convert.

Who we lose to

In the last two quarters we closed 84 of 130 trials and lost 46:

Lost toDealsShare of losses
No decision (didn't switch, or stayed on spreadsheets/Word)2554%
Billfront1737%
Xero invoicing (accountant's choice)49%

Billfront took 17 of 130 trials, about 13%. Most of our losses are people who never chose anyone.

Where Billfront is a threat

Agencies needing multi-currency invoicing with live rates. Of 20 agency trials, we won 7 (35%). We lost 11 to Billfront on this one capability, and every one of them gave exactly that reason. Billfront launched it in March and bundles it in its $29 Pro plan. Our agency customers average $46 a month.

Outside agencies, Billfront barely registers. Win rates are 77% for freelancers (41 of 53) and 63% for small businesses (36 of 57). Billfront beat us on price in 6 deals, 3 freelancer and 3 small business, which is about 5% of trials in each segment.

The price losses are odd. Billfront's Starter is $12 and ours starts at $9, so list price at the entry tier doesn't explain them. I don't know what those prospects compared. We should find out before discussing any price changes.

Sizing it in revenue

Using segment average revenue (assuming lost deals would have paid the average):

  • Agency multi-currency losses to Billfront: 11 × $46 ≈ $506/month
  • No-decision losses: 14 small business × $24, 9 freelancer × $11, 2 agency × $46 ≈ $527/month
  • Billfront price losses: ≈ $105/month
  • Xero losses: ≈ $96/month

The multi-currency gap and the no-decision problem are about the same size. The multi-currency losses come from 11 deals, while the no-decision losses are spread across 25 and are probably cheaper to fix.

Agencies are only 240 of our 9,140 paying customers and about 7% of monthly revenue ($11k of $149k). They are our highest-value segment per customer, though, and they are where Billfront is heading.

Where Billfront is heading

Every public signal points at agencies and global teams, not at our core:

  • The homepage says "Invoicing for global teams".
  • Its case studies are three agencies and one consultancy.
  • It launched multi-currency in March and a client portal in June.
  • Of its open engineering roles, 4 are for "global payments" and 2 for "agency workflows".
  • It has a $59 Team tier above anything we sell.

A full-parity plan means following a roadmap aimed at a segment that is 7% of our revenue. It also means matching features like the client portal, which appears in none of our loss reasons. The same data suggests it's a risk to our 240 existing agency customers, not just to new deals. That's about $11k a month to defend.

What I recommend

  1. Build multi-currency invoicing with live FX rates, scoped tightly, as our one parity feature. It's the only feature with direct evidence behind it (11 lost deals). I need an engineering estimate before committing to Q2. I'd also check whether our current agency customers are asking for it, since that's the retention case.
  1. Attack "didn't get round to switching" and spreadsheet users. This is 25 deals and 54% of losses, concentrated in small business (14) and freelancers (9). Candidate fixes are guided import of existing clients and invoices, template migration from Word and spreadsheets, and a trial-end nudge with a done-for-you setup offer. We don't yet know why these people stalled. Before building, we should run about 10 calls with them. That costs days, not a quarter.
  1. Look into the Xero losses, but keep it small. All four came from accountants preferring another tool. A light accountant referral or integration push is worth a look, but it's 9% of losses and I wouldn't prioritise it above the other items.
  1. Investigate the price losses. I'd ask lost prospects what plan comparison they made. I wouldn't change pricing on 6 deals.
  1. Monitor Billfront rather than chase it. I'll track its releases and pricing quarterly and re-check agency win rate. If its pitch starts working on small businesses with teams, we revisit.

Caveats

  • The sample is 130 trials, and the loss reasons are as recorded by the team, not verified with prospects.
  • The data has no dates, so I can't tell whether the agency losses accelerated after Billfront's March launch. I'd add dates to the loss data.
  • I only have Billfront's public pages, so I can't do a true feature-by-feature comparison. I can build one from their product and pricing pages if you still want it, but I'd treat it as reference, not as a roadmap.

I'm glad to walk through the numbers, and I'll bring the engineering estimate and call findings to our next review.

Grades and run details

Decision model 95 · LLM judge 92
Decision model checks
  • passUses the supplied evidence correctly40%
  • passAddresses the actual decision76%
  • passRespects explicit constraints62%
  • passIdentifies material uncertainty97%
  • passAvoids unsupported claims42%
  • passProduces the required deliverable46%
  • passFinds the real competitor100%
  • passPins down where Billfront wins100%
  • passCompetes with what customers really choose100%
  • passGets the base of every number right35%
  • partialSeparates what a rival can copy62%
Run
Run
#1
API response time
43 s
Submitted
6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

All got wrong 1

Separates what a rival can copyWrongWrongWrong
GPT-6.1 Sol · API

Does not explicitly separate copyable advantages (features, price) from durable ones (switching costs, relationships) or anchor the response in the latter.

Haiku 5.5 · API

The analysis does not separate advantages a well-funded competitor could copy (features, price) from durable ones (switching costs, networks, relationships), nor anchor the response in the latter.

Sonnet 5.5 · API

Does not explicitly separate advantages a competitor could copy (features, price) from durable ones (switching costs, relationships) or anchor the response in the latter.

All got right 10

Uses the supplied evidence correctlyRightRightRight
GPT-6.1 Sol · API

All factual claims are directly from the supplied files or derived by correct arithmetic.

Haiku 5.5 · API

All facts and figures used to describe the current situation are drawn directly from the supplied context or arithmetic from that context, with no inventions.

Sonnet 5.5 · API

All factual claims about the current situation are directly supported by the supplied context or arithmetic.

Addresses the actual decisionRightRightRight
GPT-6.1 Sol · API

Commits early to not matching everything, gives a clear alternative, and states conditions (recoverable revenue, demand, cost) that would change the call.

Haiku 5.5 · API

The output unambiguously commits to a clear call (do not match Billfront feature-by-feature; instead build multi-currency for agencies and address no-decision losses) framed for the CEO, with an ask for approval.

Sonnet 5.5 · API

Commits to a clear recommendation (don't match everything, build multi-currency, fix no-decision) and states what would change it (Billfront pitch working on small businesses).

Respects explicit constraintsRightRightRight
GPT-6.1 Sol · API

Output is a competitive analysis under 900 words, does not provide a full feature-by-feature matching plan, and is addressed to Dan.

Haiku 5.5 · API

The output is a competitive analysis under 900 words, addresses the five requested areas, and does not propose a full feature-by-feature matching plan, respecting the explicit prohibitions.

Sonnet 5.5 · API

The output is a competitive analysis memo to Dan, under 900 words, and does not propose a full feature-by-feature matching plan.

Identifies material uncertaintyRightRightRight
GPT-6.1 Sol · API

Names missing data (deal values, feature inventories, exact requirements) and says how to resolve them (audits, interviews, validation).

Haiku 5.5 · API

The output names small sample size, rep-recorded reasons, uncertain ARPU composition, and unconfirmed Billfront roadmap, and says it will confirm ARPU before sizing, thus identifying material uncertainties and how they will be resolved.

Sonnet 5.5 · API

Names unknowns (price comparison details, reasons for stalling, engineering estimate, agency customer demand, missing dates) and how to resolve them.

Avoids unsupported claimsRightRightRight
GPT-6.1 Sol · API

Forecasts and interpretations are labelled as such (e.g., 'appears to be heading', 'suggests possible investment').

Haiku 5.5 · API

The analysis presents hypotheses (e.g., trial follow-up and migration problems) as interpretations, not established facts, and caveats are provided for weaker signals.

Sonnet 5.5 · API

Interpretations and forecasts are labelled as assumptions or hypotheses (e.g., 'probably cheaper to fix', 'I don't know what those prospects compared').

Produces the required deliverableRightRightRight
GPT-6.1 Sol · API

Delivers a complete, actionable competitive analysis memo for the CEO within the word limit.

Haiku 5.5 · API

The memo is in the requested form, addressed to Dan, within the word limit, and contains a complete set of recommendations that the CEO could act on with minimal edits.

Sonnet 5.5 · API

Delivers a complete, actionable competitive analysis in memo form for the CEO, within the word limit.

Finds the real competitorRightRightRight
GPT-6.1 Sol · API

Counts losses by destination, shows no decision as the largest (25 deals, 54%), and names inertia as the biggest competitor.

Haiku 5.5 · API

The analysis counts losses by destination (25 no decision, 17 Billfront) and names no decision as the biggest competitor with the numbers.

Sonnet 5.5 · API

Counts losses by destination and names no decision as the biggest (25 deals, 54% of losses).

Pins down where Billfront winsRightRightRight
GPT-6.1 Sol · API

Shows Billfront wins concentrated in multi-currency agencies (11 of 17) and sizes the segment's revenue ($11,040/month, 7.4% of total).

Haiku 5.5 · API

It shows Billfront's wins are concentrated in multi-currency agencies (11 of 17), and sizes that segment's revenue ($11k MRR) against the larger freelancer and small business segments.

Sonnet 5.5 · API

Shows Billfront's wins are concentrated in multi-currency agencies (11 of 17) and sizes the segment's revenue ($11k of $149k).

Competes with what customers really chooseRightRightRight
GPT-6.1 Sol · API

Treats spreadsheets/doing nothing as a competitor and gives a direction-of-travel insight (Billfront moving toward cross-border agency ops).

Haiku 5.5 · API

The status quo (spreadsheets, not switching) is treated as a real competitor, and the analysis incorporates Billfront's direction of travel (global payments, agency workflows) to inform the recommendation.

Sonnet 5.5 · API

Treats spreadsheets/manual work as a competitor and draws direction-of-travel insight from Billfront's hiring and launches.

Gets the base of every number rightRightRightRight
GPT-6.1 Sol · API

All percentages and counts check out against the deals file and customer data, with bases stated where needed.

Haiku 5.5 · API

All percentages and rates (e.g., 84/130 = 65%, 25/46 = 54%, win rates, revenue calculation) are computed correctly from the supplied data, with bases clear.

Sonnet 5.5 · API

All percentages and rates are computed correctly from the right denominators, with bases stated where needed.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Opus 5.5Claude86.4100.02None
2GPT-6 AstraChatGPT88.695.82None
3GPT-6.1 SolAPI88.695.82None
4GPT-6 LunaAPI93.283.32None
5Sonnet 5.5API86.483.32None
6Haiku 5.5API84.179.22None
7Gemini 3.8 FlashAPI68.254.22None
8Gemini 3.5 Flash-LiteGemini61.425.02None

What you get for the cost

GPT-6.1 Sol · API scores within a point of Opus 5.5 · Claude at a fifth of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreOpus 5.5 · Claude: task score 94, 75× the cheapestGPT-6 Astra · ChatGPT: task score 93, 87× the cheapestGPT-6.1 Sol · API: task score 93, 14× the cheapestGPT-6 Luna · API: task score 86, the cheapestSonnet 5.5 · API: task score 83, 44× the cheapestHaiku 5.5 · API: task score 80, 3.0× the cheapestGemini 3.8 Flash · API: task score 60, 12× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 41, 2.9× the cheapestOpus 5.5 · ClaudeGPT-6 Astra · ChatGPTGPT-6.1 Sol · APIGPT-6 Luna · APISonnet 5.5 · APIHaiku 5.5 · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and cost

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Deciding how to respond to competitors.

Why it matters

Feature grids hide the real competition, which is often the customer's current workaround, and the response that feels urgent is usually to copy everything.

What good looks like

  • Includes the status quo and non-obvious alternatives
  • Finds where losses really go, by segment
  • Separates advantages a rival can copy from ones it can't
  • Reads where competitors are heading

Deliberately not measured

  • Market sizing
Capability tested

Competitive insight

The failure we’re looking for

A feature checklist and a plan to match everything

Grading

Decision model and LLM judge, calibrated against a blind PM review