Humanity’s Last PRD
Latest drop: Haiku 5.5 · 8 Oct
Tasks / Strategy

Competitor analysis

Can the model find where a competitor can actually hurt you, from win/loss and churn data, rather than list features?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 63% were usable with at most a quick edit.

Reliably right

  1. Respects explicit constraints100% pass
    Delivers a competitive analysis memo to Dan within 900 words, addressing all requested topics without proposing a full feature-by-feature match.
    Opus 5.5 · Claude · The competitor isn't who you think
  2. Reads where Crestline is heading100% pass
    Uses Crestline's hiring (payroll, multi-entity, accountant partnerships) to infer direction, flags the partner-programme risk, and adjusts the response accordingly.
    GPT-6 Astra · ChatGPT · Free forever, funded
  3. Produces the required deliverable97% pass
    The memo is complete, in the right form for the CEO, within the word limit, and actionable with light edits.
    Opus 5.5 · Claude · The competitor isn't who you think

Where it slips

  1. Uses the supplied evidence correctly56% pass
    Two factual claims are wrong: the number of unconnected multi-user accounts going to Crestline is 40, not 42, and the number of connected 6-20 cancellations is 5, not 4.
    Sonnet 5.5 · API · Free forever, funded
  2. Gets the base of every number right61% pass
    Derived figures for unconnected multi-user cancellations (42 vs actual 40) and connected 6-20 cancellations (4 vs actual 5) are incorrect.
    Sonnet 5.5 · API · Free forever, funded
  3. Separates what a rival can copy66% pass
    Does not explicitly separate advantages a well-funded competitor could copy from those it couldn't, nor anchor the response in durable advantages with evidence.
    GPT-6 Luna · API · The competitor isn't who you think

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Competes with what customers really choose

    Does the analysis treat the status quo (spreadsheets, manual work, doing nothing) as a competitor, and draw at least one implication from where competitors are investing rather than only from today's features?

    Passes when Analyses the status quo, strengths included, and gives a direction-of-travel insight that affects the recommendation.

    April DunfordLenny’s Podcast

  2. Separates what a rival can copy

    Does the analysis separate advantages a well-funded competitor could copy (features, price, being first) from ones it couldn't easily (switching costs, networks, relationships), and build the response on the latter?

    Passes when Names which advantages are copyable and which aren't, with evidence, and anchors the response in the durable ones.

    Hamilton HelmerLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for growth at Tallybook. Our CEO, Dan Whitlock, wants a feature-by-feature comparison against Billfront and a plan to match everything they have by Q2 (his note is below). Write him a competitive analysis of no more than 900 words: who we really lose to, where Billfront is a real threat and where it isn't, where it's heading, and what we should do instead of (or as well as) matching it. The files are attached.

What the model was given5 items: About Tallybook, ceo_note.md, deals_last_two_quarters.csv (trials that became deals, won or lost), customers_by_segment.md, billfront_public_info.md
About TallybookInvoicing and payments for freelancers and small businesses. Plans from $9 to $39 a month.
ceo_note.md1 lines · Download
From Dan (CEO): "Billfront is killing us. I want a full feature-by-feature comparison against them and a plan to match everything they have by Q2."
deals_last_two_quarters.csv (trials that became deals, won or lost)deal,segment,outcome,primary_reason,went_to T-900,Small business,won,, T-901,Freelancer,won,, T-902,Freelancer,won,, T-903,Small business,won,, T-904,Small business,lost,Accountant uses something else,Xero invoicing T-905,Small business,won,, T-906,Small business,lost,Didn't get round to switching,No decision T-907,Freelancer,won,, T-908,Freelancer,won,, T-909,Small business,won,, T-910,Freelancer,lost,Price,Billfront T-911,Freelancer,won,, T-912,Small business,lost,Price,Billfront T-913,Freelancer,won,, T-914,Freelancer,won,, T-915,Small business,won,, T-916,Freelancer,won,, T-917,Small business,won,, T-918,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-919,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-920,Freelancer,won,, T-921,Agency (multi-currency),won,, T-922,Small business,won,, T-923,Small business,won,, T-924,Small business,won,, T-925,Freelancer,won,, T-926,Freelancer,won,, T-927,Freelancer,won,, T-928,Small business,won,, T-929,Small business,lost,Didn't get round to switching,No decision T-930,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-931,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-932,Small business,won,, T-933,Small business,won,, T-934,Small business,won,, T-935,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-936,Agency (multi-currency),lost,Didn't get round to switching,No decision T-937,Small business,lost,Didn't get round to switching,No decision T-938,Agency (multi-currency),won,, T-939,Small business,lost,Didn't get round to switching,No decision T-940,Small business,won,, T-941,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-942,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-943,Freelancer,won,, T-944,Small business,won,, T-945,Small business,lost,Accountant uses something else,Xero invoicing T-946,Small business,won,, T-947,Small business,won,, T-948,Small business,won,, T-949,Small business,lost,Didn't get round to switching,No decision T-950,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-951,Freelancer,won,, T-952,Freelancer,won,, T-953,Freelancer,won,, T-954,Agency (multi-currency),won,, T-955,Small business,lost,Didn't get round to switching,No decision T-956,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-957,Freelancer,won,, T-958,Small business,won,, T-959,Freelancer,won,, T-960,Small business,lost,Didn't get round to switching,No decision T-961,Freelancer,won,, T-962,Small business,won,, T-963,Agency (multi-currency),won,, T-964,Small business,won,, T-965,Small business,won,, T-966,Small business,won,, T-967,Small business,lost,Didn't get round to switching,No decision T-968,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-969,Small business,won,, T-970,Freelancer,lost,Price,Billfront T-971,Small business,lost,Didn't get round to switching,No decision T-972,Small business,lost,Accountant uses something else,Xero invoicing T-973,Small business,lost,Accountant uses something else,Xero invoicing T-974,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-975,Small business,lost,Price,Billfront T-976,Freelancer,won,, T-977,Small business,lost,Didn't get round to switching,No decision T-978,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-979,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-980,Freelancer,won,, T-981,Small business,won,, T-982,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-983,Small business,won,, T-984,Small business,lost,Didn't get round to switching,No decision T-985,Freelancer,won,, T-986,Freelancer,won,, T-987,Small business,lost,Didn't get round to switching,No decision T-988,Small business,won,, T-989,Freelancer,won,, T-990,Freelancer,won,, T-991,Freelancer,won,, T-992,Agency (multi-currency),won,, T-993,Freelancer,won,, T-994,Freelancer,won,, T-995,Small business,won,, T-996,Small business,won,, T-997,Small business,won,, T-998,Small business,lost,Didn't get round to switching,No decision T-999,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-1000,Agency (multi-currency),won,, T-1001,Small business,lost,Price,Billfront T-1002,Small business,won,, T-1003,Freelancer,won,, T-1004,Agency (multi-currency),lost,Didn't get round to switching,No decision T-1005,Freelancer,won,, T-1006,Small business,won,, T-1007,Freelancer,won,, T-1008,Small business,won,, T-1009,Freelancer,lost,Price,Billfront T-1010,Small business,won,, T-1011,Freelancer,won,, T-1012,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-1013,Freelancer,won,, T-1014,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-1015,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-1016,Freelancer,won,, T-1017,Freelancer,won,, T-1018,Freelancer,won,, T-1019,Small business,lost,Didn't get round to switching,No decision T-1020,Freelancer,won,, T-1021,Small business,won,, T-1022,Agency (multi-currency),won,, T-1023,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-1024,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-1025,Freelancer,won,, T-1026,Freelancer,won,, T-1027,Freelancer,won,, T-1028,Small business,won,, T-1029,Freelancer,won,,
customers_by_segment.md7 lines · Download
# Tallybook customers and revenue (September)

| Segment | Paying customers | Average revenue per month |
|---|---|---|
| Freelancer | 5,800 | $11 |
| Small business | 3,100 | $24 |
| Agency (multi-currency) | 240 | $46 |
billfront_public_info.md6 lines · Download
# Billfront (public pages, September)

- Pricing: $12/month Starter, $29/month Pro (multi-currency, live FX rates), $59/month Team.
- Homepage: "Invoicing for global teams". Case studies: three agencies, one consultancy.
- Recent launches: multi-currency invoicing (March), client portal (June).
- Job posts: 4 engineers for "global payments", 2 for "agency workflows".
What a strong answer doesThe answer key the graders mark against

Shows from the deals file that the biggest competitor is no decision: of 46 lost deals, 25 ended with the prospect staying on spreadsheets or never switching, 17 went to Billfront and 4 to Xero invoicing. Billfront's wins are concentrated in multi-currency agencies (11 of its 17), on a specific gap (invoicing in several currencies with live rates); elsewhere it won only 6 deals, mostly on price. Agencies are a small segment (240 customers at about $46 a month, about $11,000 MRR, against about $64,000 from freelancers and $74,000 from small businesses), and Billfront is heading further into them (global payments and agency workflow hiring). Recommends against matching everything: fix switching friction for small businesses who 'didn't get round to it' (imports, setup help, accountant tie-ins), make a deliberate call on multi-currency (build it if agencies matter strategically, or cede the segment), and keep the advantages Billfront can't easily copy. Gives a short comparison only where it matters.

Critical failures (cap the score at 40)
  • Recommends matching Billfront feature for feature
Task

Mid-level · v1.0

The outputs

8 setups answered this brief.

Opus 5.5Claude98Usable with a quick editGPT-6 LunaAPI94Usable with a quick editGPT-6 AstraChatGPT94Usable with a quick editSonnet 5.5API94Usable with a quick editGPT-6.1 SolAPI94Usable with a quick editHaiku 5.5API92Usable with a quick editGemini 3.8 FlashAPI71Usable with a quick editGemini 3.5 Flash-LiteGemini35Needs more work
Gemini 3.8 FlashAPI

Usable with a quick edit6 of 11 checks passed

The output is well-structured, evidence is correctly used, and the recommendation is clear. A PM could use it with minor wording edits (e.g., softening the 'vacating' claim) without re-checking data or redoing reasoning.

Output

MEMORANDUM

TO: Dan Whitlock, CEO FROM: PM for Growth DATE: October 14, 2023 SUBJECT: Competitive Analysis: Billfront & Win/Loss Findings

---

Executive Summary

Billfront is not killing us. Across the last two quarters (130 evaluated deals), our win rate stands at 65% (84 wins).

Of the 46 deals lost, 54% (25 deals) were lost to inertia and the status quo (“No decision”), not a competitor. Billfront won 17 deals (37% of losses), but 65% of those losses were concentrated in a single niche: cross-border agencies needing live multi-currency rates.

Matching Billfront feature-for-feature by Q2 would misallocate our roadmap to chase a customer segment that accounts for just 7.4% of our revenue, while ignoring the friction costing us half of our lost pipeline in our core market.

---

1. Who We Really Lose To

Our win/loss data over the last two quarters paints a very clear picture of why prospective customers fail to convert:

``` Loss Reasons Across All Segments (46 Lost Deals) ┌───────────────────────────────────────────────┬───────┬───────┐ │ Competitor / Outcome │ Deals │ Share │ ├───────────────────────────────────────────────┼───────┼───────┤ │ No Decision: "Didn't get round to switching" │ 16 │ 35% │ │ Billfront: Multi-currency FX gap │ 11 │ 24% │ │ No Decision: Stayed on Spreadsheets / Word │ 9 │ 20% │ │ Billfront: Price │ 6 │ 13% │ │ Xero: Accountant preference │ 4 │ 9% │ └───────────────────────────────────────────────┴───────┴───────┘ ```

  • #1 Competitor: Status Quo & Inertia (25 deals / 54% of losses). Small businesses abandon onboarding because switching is too painful (14 deals), and freelancers default back to Word/Excel templates (9 deals).
  • #2 Competitor: Billfront (17 deals / 37% of losses). 11 of these were multi-currency agencies; only 6 were core domestic freelancers or small businesses citing price.
  • #3 Competitor: Accounting Ecosystem Lock-in (4 deals / 9% of losses). Small businesses whose external accountants mandate Xero.

---

2. Where Billfront Is (and Isn't) a Threat

To evaluate Billfront’s threat level, we must cross-reference our losses against where our revenue actually comes from:

Customer SegmentPaying CustomersARPUMonthly Revenue% of Total MRR
:---:---:---:---:---
Freelancer5,800$11$63,80042.7%
Small Business3,100$24$74,40049.9%
Agency (Multi-Currency)240$46$11,0407.4%
Total9,140—$149,240100%

#### Where Billfront is a real threat: * Multi-Currency Agencies: Billfront launched multi-currency invoicing with live FX in March and is winning on that capability. We lost 11 agency deals specifically on this feature. If an agency invoices international clients in varying currencies, we currently do not compete.

#### Where Billfront is NOT a threat: * Freelancers and Domestic SMBs: These two groups generate 92.6% of our revenue ($138k MRR). Out of 110 pipeline trials in these segments, Billfront took only 6 deals (5.5%). * Billfront’s starting tier is $12/month (vs. our $9/month). The 6 price losses suggest confusion around feature gating between our tiers rather than Billfront undercutting our baseline price.

---

3. Where Billfront Is Heading

Billfront’s public positioning and hiring roadmap reveal an explicit upmarket strategy: * Positioning: Rebranded to "Invoicing for global teams", with agency-only case studies. * Recent Releases: Client portal (June), multi-currency (March). * Hiring: Actively recruiting 4 engineers for "global payments" and 2 for "agency workflows".

Billfront is vacating the simple domestic SMB and freelancer market to become specialized back-office software for distributed, cross-border agencies. Chasing them feature-for-feature means chasing them into an enterprise agency tier.

---

4. Strategic Recommendation: What We Should Do Instead

Matching Billfront’s entire product by Q2 would consume all engineering resources on agency workflows (portals, complex multi-currency ledgering, global payments) for our smallest segment, while leaving our 93% revenue engine vulnerable to churn and onboarding friction.

Instead, I propose a focused, dual-track plan for Q1–Q2:

#### Pillar 1: Attack the 54% "No Decision" Leak (Growth Priority) Reducing activation friction will yield 3x the revenue impact of chasing agency features: 1. Frictionless Migration Tooling: Build 1-click CSV/spreadsheet import for customer lists and open invoices to eliminate the "didn't get round to switching" barrier for SMBs. 2. Template-to-App Onboarding: Create lightweight web invoice generators that let spreadsheet users experience Tallybook’s value before forcing account creation. 3. Accountant Collaboration View: Introduce a basic accountant export/read-only seat to stop leakage to Xero.

#### Pillar 2: Selective Multi-Currency Response (Monetization Priority) We should not build custom "agency workflows," but we should neutralize the specific product gap causing deal losses: 1. Targeted Multi-Currency & Live FX: Integrate automated FX exchange rates for cross-border invoicing by Q2. 2. Package as a Premium Tier / Add-on: Billfront charges $29/mo (Pro) and $59/mo (Team) for this. We can package multi-currency into a dedicated $49/mo "International Tier" or a paid add-on, immediately lifting agency ARPU without bloating the product for domestic users.

#### Pillar 3: Audit Pricing Tiers Investigate the 6 lost deals to Billfront on price. Ensure our $9 entry plan is clearly differentiated and does not wall off basic capabilities that drive price-sensitive users away.

---

Next Steps

I recommend holding off on an across-the-board feature-parity initiative. I will deliver a detailed spec and ROI projection by Friday for (1) Migration/Switching Flow Optimization and (2) Multi-Currency Invoicing as a Premium Add-on for Q1 delivery.

Claims the judge couldn’t find in the brief

  • Billfront is vacating the simple domestic SMB and freelancer market to become specialized back-office software for distributed, cross-border agencies.

Grades and run details

Decision model 73 · LLM judge 67
Decision model checks
  • failUses the supplied evidence correctly20%
  • passAddresses the actual decision41%
  • passRespects explicit constraints35%
  • partialIdentifies material uncertainty46%
  • failAvoids unsupported claims25%
  • passProduces the required deliverable70%
  • passFinds the real competitor96%
  • passPins down where Billfront wins100%
  • passCompetes with what customers really choose100%
  • passGets the base of every number right13%
  • partialSeparates what a rival can copy65%
Run
Run
#1
API response time
26 s
Submitted
6 Oct 2026
Opus 5.5Claude

Usable with a quick edit11 of 11 checks passed

The analysis is data-backed, complete, and directly actionable. A PM could use it with minor wording edits and no need to re-verify evidence or reasoning.

Output

To: Dan Whitlock From: Growth PM Re: Billfront: where they actually hurt us, and what to do about it

---

The short version

Billfront isn't killing us. It is beating us in one segment, for one reason. Matching its whole feature set by Q2 would spend most of our roadmap on customers who aren't leaving us for Billfront. My recommendation is to build one targeted feature (multi-currency invoicing with live rates), deliberately skip the rest, and put equal effort into our real biggest loss: prospects who never switch to anything.

Who we actually lose to

Over the last two quarters, 130 trials became deals. We won 84 (65%) and lost 46.

Lost toDealsShare of lossesMain reason
No decision2554%Didn't get round to switching (16), stayed on spreadsheets/Word (9)
Billfront1737%Multi-currency with live rates (11), price (6)
Xero invoicing49%Their accountant uses something else

Our biggest competitor is inertia, not Billfront. Billfront accounts for 13% of all trials (17 of 130).

Where Billfront is a real threat, and where it isn't

It's a real threat in agencies. We lose 13 of 20 agency deals (35% win rate). Eleven of those losses went to Billfront, every one for the same reason: invoicing in several currencies with live rates. This is a single, clear feature gap, and it is the core of Billfront's paid offer. Their $29 Pro plan is built around it.

It isn't a threat in our core. Freelancers (77% win rate) and small businesses (63%) make up 92% of our revenue. Billfront took 6 deals from these segments, all on price. That is roughly 5% of core-segment deals.

The price losses are puzzling. Our entry plan is $9 and Billfront's is $12, so "price" probably means something else. Most likely a prospect needed a feature that's on our higher tier and is cheaper at Billfront, or compared our mid plan to their Starter. We should pull those six trial records before reacting to them.

How much revenue is at stake (each lost deal valued at its segment's average monthly revenue):

  • Billfront losses: about $610/month (multi-currency ~$505, price ~$105)
  • No-decision losses: about $525/month
  • Xero losses: about $95/month

By revenue, Billfront and inertia are roughly level. The difference is the fix. The Billfront gap is one feature. Inertia is spread across our largest segments and touches every trial.

Where Billfront is heading

Their public signals all point the same way, away from our core:

  • Positioning: the homepage says "Invoicing for global teams." Their case studies are three agencies and a consultancy, with no freelancers or local small businesses.
  • Launches: multi-currency in March, a client portal in June. Both are agency features.
  • Hiring: 4 engineers for "global payments" and 2 for "agency workflows."
  • Pricing: their plans go up to $59/month for teams. Our average agency customer pays us $46.

Billfront is moving upmarket into international agencies and cross-border payments. The risk to watch is global payments. If they make it easy to get paid across currencies, not just invoice in them, that could eventually appeal to our freelancers with overseas clients. For now there's no sign of that in our deal data.

What I recommend instead of matching everything

1. Build multi-currency invoicing with live rates for Q2 (match this one feature). This is the only Billfront feature that shows up in our losses, and it explains nearly all of them. Agencies are our highest-value customers per head ($46/month vs $24 and $11), even though the segment is small (240 customers, about 7% of revenue). Closing this gap protects them and blocks Billfront's main way in. Scope it tightly: multiple invoice currencies and live rates. Leave payments for later.

2. Don't build the client portal, team tier, or global payments yet. No lost deal cites any of them. Building them means chasing Billfront's strategy rather than our customers. We should re-check this every quarter against new loss data and their hiring.

3. Attack "no decision" with the same priority as #1. This is 25 of 46 losses. Two specific fixes: - Spreadsheet/Word import for freelancers (9 losses). Let them upload their existing template or client list and send a first invoice in minutes. - Switching help for small businesses (14 losses). Offer guided migration, "import your last 10 invoices," and trial nudges tied to their next billing date. Because these improve every trial, they also help us in the segments that make up most of our revenue.

4. Small bet on Xero (4 losses). Check whether a Xero sync, or simply marketing the one we have, gives accountants a reason to say yes.

5. Diagnose the six price losses before changing pricing.

Caveats

This is 130 deals over two quarters, with one recorded reason per loss. The revenue figures are rough, monthly, and per deal. The direction is clear enough to act on, but we should keep tagging loss reasons and re-run this analysis in a quarter.

If it helps, I can turn #1 and #3 into a Q2 roadmap proposal with effort estimates.

Grades and run details

Decision model 95 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly20%
  • passAddresses the actual decision88%
  • passRespects explicit constraints57%
  • passIdentifies material uncertainty81%
  • passAvoids unsupported claims33%
  • passProduces the required deliverable80%
  • passFinds the real competitor100%
  • passPins down where Billfront wins96%
  • passCompetes with what customers really choose100%
  • passGets the base of every number right33%
  • partialSeparates what a rival can copy46%
Run
Run
#1
Time to output
46 s
Submitted
5 Oct 2026
GPT-6 AstraChatGPT

Usable with a quick edit10 of 11 checks passed

The analysis is evidence-based and actionable; a PM could add a sentence on durable advantages without redoing any reasoning or re-checking evidence.

Output

To: Dan Whitlock Subject: Q2 competitive plan—beat the reasons we lose, not Billfront’s feature list

Dan—

Billfront is a real threat in multi-currency agencies, but the deal data does not support “Billfront is killing us” across our business. I recommend a Q2 plan focused on switching friction in our core market, with a gated investment in multi-currency—not blanket feature parity.

1. Who we actually lose to

Across 130 trials with recorded outcomes over the last two quarters, we won 84 and lost 46.

DestinationLost dealsShare of lossesRecorded reason
------:---:---
No decision2554%16 never switched; 9 stayed with spreadsheets/Word
Billfront1737%11 multi-currency/live FX; 6 price
Xero invoicing49%Accountant uses another system

Our largest competitor is inertia. More deals stall or retain manual workflows than go to Billfront.

The segment split matters:

  • Freelancers and small businesses: 110 opportunities, 77 wins, and only six losses to Billfront—all attributed to price.
  • Multi-currency agencies: 20 opportunities, seven wins, 11 losses to Billfront, and two no-decisions. Billfront captures 55% of these opportunities and 85% of their losses.

These are counts from recorded trial outcomes—not market share, churn, or revenue-weighted losses. We lack deal values and quarter-by-quarter breakdowns, so cannot conclude that competitive losses are accelerating.

2. Where Billfront threatens us—and where it doesn’t

Our installed base remains overwhelmingly freelancers and small businesses: 97% of paying customers and 93% of monthly revenue, using September customer counts and ARPU. Agencies contribute approximately $11,040 of $149,240 monthly revenue.

Billfront is therefore a concentrated segment threat, not yet a demonstrated broad-based threat to our core. However, agencies represent 15% of this opportunity sample versus 3% of current customers. Ignoring them could constrain an emerging growth segment.

The defensible comparison from the supplied evidence is:

DimensionBillfrontTallybook / implication
Subscription pricing$12 Starter; $29 Pro; $59 Team$9–$39 plans. Billfront is not universally cheaper; investigate the actual packages compared in six price losses.
Multi-currency and live FXIncluded in Pro; multi-currency launched in MarchEleven losses establish an unmet customer requirement. This is the strongest case for targeted parity.
Client portalLaunched in JuneTallybook capability is unspecified. No recorded loss cites it; validate demand before prioritizing.
Team/agency capabilitiesTeam tier; agency-focused hiringDetailed functionality is unknown. Neither the plan name nor hiring proves specific feature advantages.
Positioning“Invoicing for global teams”; three agency and one consultancy case studiesTallybook serves freelancers and small businesses. Their positioning overlaps most strongly with our agency segment.

A complete feature-by-feature audit requires product access and our own capability inventory. We should do a bounded audit, but not turn every discovered difference into a roadmap commitment.

3. Where Billfront is heading

The evidence points toward cross-border payments and agency workflows: multi-currency is already shipped, four engineering roles target global payments, and two target agency workflows. The portal could support deeper agency-client collaboration.

That suggests a specialist competitor becoming stronger in international service businesses—not necessarily a competitor optimizing for solo freelancers. Hiring indicates direction, not a confirmed launch schedule. Matching today’s list could leave us following their next list while neglecting our larger conversion problem.

4. What we should do by Q2

First: make switching easier for our core customers. Interview recent no-decision prospects, then test guided setup, spreadsheet/client imports, reusable invoice templates, and assisted migration where the workflow audit shows gaps. Sell the outcome—getting the first invoice sent and paid with less effort—not feature volume. Measure trial-to-paid conversion, time to first invoice, first payment, and no-decision losses against a baseline or control.

Second: gate a focused multi-currency investment. Validate requirements with lost agencies and our seven agency wins: supported currencies, live-rate handling, settlement needs, and willingness to pay. Estimate build, provider, compliance, and support costs. Commit to a scoped Q2 release only if expected incremental contribution and retention justify it versus core activation work. Do not bundle a portal or full Team suite into that decision.

Third: diagnose price before cutting it. Review the six price losses for required features, chosen Billfront tier, discounts, and perceived value. Test clearer packaging or targeted offers rather than an across-the-board reduction. Reconcile agency ARPU of $46 with our stated $39 plan ceiling before modeling returns.

Fourth: reduce accountant-driven exclusions. Investigate the four Xero losses and test whether accountant-friendly exports or interoperability can preserve Tallybook as the invoicing layer.

Decision requested: replace “match everything by Q2” with a measured conversion program and a gated multi-currency bet. Keep a monthly competitor review, but fund work according to customer demand and incremental economics—not checklist completeness.

Grades and run details

Decision model 95 · LLM judge 92
Decision model checks
  • passUses the supplied evidence correctly44%
  • passAddresses the actual decision85%
  • passRespects explicit constraints55%
  • passIdentifies material uncertainty99%
  • passAvoids unsupported claims64%
  • passProduces the required deliverable48%
  • passFinds the real competitor100%
  • passPins down where Billfront wins100%
  • passCompetes with what customers really choose100%
  • passGets the base of every number right36%
  • partialSeparates what a rival can copy56%
Run
Run
#1
Time to output
44 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 5

Uses the supplied evidence correctlyMixedRightRight
Gemini 3.8 Flash · API

All factual claims about the current situation are taken correctly from the supplied context or derived by accurate arithmetic; the one unsupported claim is a forecast, not a current-situation fact.

Opus 5.5 · Claude

All factual claims about the current situation are directly supported by the supplied context or arithmetic from it.

GPT-6 Astra · ChatGPT

All facts and figures are correctly taken from the supplied data, with no invented information about the current situation.

Addresses the actual decisionMixedRightRight
Gemini 3.8 Flash · API

The output commits to a clear recommendation (don't match everything, do Pillars 1-3) but does not state what result or condition would change that call, as required.

Opus 5.5 · Claude

Commits early to a clear recommendation (build multi-currency, skip the rest, attack no decision) and states that re-checking quarterly against new data would change it.

GPT-6 Astra · ChatGPT

Commits clearly to not matching everything, instead recommending a switching-friction program and a gated multi-currency investment, with conditions that would change the call.

Identifies material uncertaintyWrongRightRight
Gemini 3.8 Flash · API

The output does not name specific unknowns that could change the decision, nor say how they would be resolved or what would change the call.

Opus 5.5 · Claude

Names specific unknowns (true reason for price losses, future global payments risk) and says to pull trial records and re-run analysis quarterly.

GPT-6 Astra · ChatGPT

Names missing data (deal values, quarter breakdowns) and conditions for the multi-currency investment, and says how to resolve them through interviews, tests, and validation.

Avoids unsupported claimsWrongRightRight
Gemini 3.8 Flash · API

The claim that 'Billfront is vacating the simple domestic SMB and freelancer market' is presented as an established fact, but the evidence only shows they are targeting agencies, not that they are leaving other markets.

Opus 5.5 · Claude

Hypotheses about price losses are clearly labelled as 'probably' and 'most likely'; forecasts are presented as risks to watch, not facts.

GPT-6 Astra · ChatGPT

Interpretations and forecasts are labelled as suggestions or conclusions from data, not presented as established fact.

Separates what a rival can copyWrongRightWrong
Gemini 3.8 Flash · API

The output does not explicitly name which advantages are copyable and which are durable, nor anchor the response in the latter, as required.

Opus 5.5 · Claude

Separates the copyable feature gap (multi-currency) from the durable advantage of reducing switching friction for core customers, and anchors the response in the latter.

GPT-6 Astra · ChatGPT

Does not explicitly separate which advantages are copyable by a well-funded competitor and which are durable, nor anchor the response in the durable ones.

All got right 6

Respects explicit constraintsRightRightRight
Gemini 3.8 Flash · API

The output is a competitive analysis memo under 900 words, addressed to the CEO, and does not propose a full feature-by-feature matching plan, respecting the brief's request to say what to do instead.

Opus 5.5 · Claude

Delivers a competitive analysis memo to Dan within 900 words, addressing all requested topics without proposing a full feature-by-feature match.

GPT-6 Astra · ChatGPT

The output is a competitive analysis under 900 words, addressed to the CEO, and proposes what to do instead of matching, respecting the brief's request.

Produces the required deliverableRightRightRight
Gemini 3.8 Flash · API

The memo is complete, within 900 words, addressed to Dan, and provides a usable competitive analysis with a clear recommendation.

Opus 5.5 · Claude

The memo is complete, in the right form for the CEO, within the word limit, and actionable with light edits.

GPT-6 Astra · ChatGPT

The memo is complete, within the word limit, and directly usable by the CEO with actionable recommendations.

Finds the real competitorRightRightRight
Gemini 3.8 Flash · API

The analysis counts losses by destination, shows 25 no-decision losses vs 17 to Billfront, and names no decision as the biggest competitor.

Opus 5.5 · Claude

Counts 25 no-decision losses vs 17 Billfront, names inertia as the biggest competitor, and provides the numbers.

GPT-6 Astra · ChatGPT

Counts 25 no-decision losses versus 17 to Billfront and names inertia as the largest competitor.

Pins down where Billfront winsRightRightRight
Gemini 3.8 Flash · API

It shows Billfront's wins are concentrated in multi-currency agencies (11 of 17) and sizes the agency segment at 7.4% of MRR against freelancer and small business.

Opus 5.5 · Claude

Shows Billfront's wins are concentrated in multi-currency agencies (11 of 17) and sizes the segment at 240 customers, $46/month, ~7% of revenue.

GPT-6 Astra · ChatGPT

Shows Billfront's wins are concentrated in multi-currency agencies (11 of 17) and sizes the segment's revenue at $11k of $149k.

Competes with what customers really chooseRightRightRight
Gemini 3.8 Flash · API

The analysis treats spreadsheets and inertia as a competitor, and draws a direction-of-travel insight (Billfront moving upmarket) that affects the recommendation.

Opus 5.5 · Claude

Treats spreadsheets/manual work/no decision as a competitor and draws a direction-of-travel insight from Billfront's hiring and launches to recommend against chasing their roadmap.

GPT-6 Astra · ChatGPT

Treats the status quo (spreadsheets, no decision) as a competitor and draws an implication from Billfront's hiring direction.

Gets the base of every number rightRightRightRight
Gemini 3.8 Flash · API

All derived percentages and rates use the correct denominators (130 deals, 46 losses, 110 freelancer+small business trials) and check out against the data.

Opus 5.5 · Claude

All percentages, win rates, and revenue figures are computed from the correct denominators and match the supplied data.

GPT-6 Astra · ChatGPT

All percentages are correctly computed with clear denominators stated or implied.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Opus 5.5Claude86.4100.02None
2GPT-6 AstraChatGPT88.695.82None
3GPT-6.1 SolAPI88.695.82None
4GPT-6 LunaAPI93.283.32None
5Sonnet 5.5API86.483.32None
6Haiku 5.5API84.179.22None
7Gemini 3.8 FlashAPI68.254.22None
8Gemini 3.5 Flash-LiteGemini61.425.02None

What you get for the cost

GPT-6.1 Sol · API scores within a point of Opus 5.5 · Claude at a fifth of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreOpus 5.5 · Claude: task score 94, 75× the cheapestGPT-6 Astra · ChatGPT: task score 93, 87× the cheapestGPT-6.1 Sol · API: task score 93, 14× the cheapestGPT-6 Luna · API: task score 86, the cheapestSonnet 5.5 · API: task score 83, 44× the cheapestHaiku 5.5 · API: task score 80, 3.0× the cheapestGemini 3.8 Flash · API: task score 60, 12× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 41, 2.9× the cheapestOpus 5.5 · ClaudeGPT-6 Astra · ChatGPTGPT-6.1 Sol · APIGPT-6 Luna · APISonnet 5.5 · APIHaiku 5.5 · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and cost

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Deciding how to respond to competitors.

Why it matters

Feature grids hide the real competition, which is often the customer's current workaround, and the response that feels urgent is usually to copy everything.

What good looks like

  • Includes the status quo and non-obvious alternatives
  • Finds where losses really go, by segment
  • Separates advantages a rival can copy from ones it can't
  • Reads where competitors are heading

Deliberately not measured

  • Market sizing
Capability tested

Competitive insight

The failure we’re looking for

A feature checklist and a plan to match everything

Grading

Decision model and LLM judge, calibrated against a blind PM review