Humanity’s Last PRD
Latest drop: Haiku 5.5 · 8 Oct
Tasks / Strategy

Competitor analysis

Can the model find where a competitor can actually hurt you, from win/loss and churn data, rather than list features?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 63% were usable with at most a quick edit.

Reliably right

  1. Respects explicit constraints100% pass
    Delivers a competitive analysis memo to Dan within 900 words, addressing all requested topics without proposing a full feature-by-feature match.
    Opus 5.5 · Claude · The competitor isn't who you think
  2. Reads where Crestline is heading100% pass
    Uses Crestline's hiring (payroll, multi-entity, accountant partnerships) to infer direction, flags the partner-programme risk, and adjusts the response accordingly.
    GPT-6 Astra · ChatGPT · Free forever, funded
  3. Produces the required deliverable97% pass
    The memo is complete, in the right form for the CEO, within the word limit, and actionable with light edits.
    Opus 5.5 · Claude · The competitor isn't who you think

Where it slips

  1. Uses the supplied evidence correctly56% pass
    Two factual claims are wrong: the number of unconnected multi-user accounts going to Crestline is 40, not 42, and the number of connected 6-20 cancellations is 5, not 4.
    Sonnet 5.5 · API · Free forever, funded
  2. Gets the base of every number right61% pass
    Derived figures for unconnected multi-user cancellations (42 vs actual 40) and connected 6-20 cancellations (4 vs actual 5) are incorrect.
    Sonnet 5.5 · API · Free forever, funded
  3. Separates what a rival can copy66% pass
    Does not explicitly separate advantages a well-funded competitor could copy from those it couldn't, nor anchor the response in durable advantages with evidence.
    GPT-6 Luna · API · The competitor isn't who you think

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Competes with what customers really choose

    Does the analysis treat the status quo (spreadsheets, manual work, doing nothing) as a competitor, and draw at least one implication from where competitors are investing rather than only from today's features?

    Passes when Analyses the status quo, strengths included, and gives a direction-of-travel insight that affects the recommendation.

    April DunfordLenny’s Podcast

  2. Separates what a rival can copy

    Does the analysis separate advantages a well-funded competitor could copy (features, price, being first) from ones it couldn't easily (switching costs, networks, relationships), and build the response on the latter?

    Passes when Names which advantages are copyable and which aren't, with evidence, and anchors the response in the durable ones.

    Hamilton HelmerLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for growth at Tallybook. Our CEO, Dan Whitlock, wants a feature-by-feature comparison against Billfront and a plan to match everything they have by Q2 (his note is below). Write him a competitive analysis of no more than 900 words: who we really lose to, where Billfront is a real threat and where it isn't, where it's heading, and what we should do instead of (or as well as) matching it. The files are attached.

What the model was given5 items: About Tallybook, ceo_note.md, deals_last_two_quarters.csv (trials that became deals, won or lost), customers_by_segment.md, billfront_public_info.md
About TallybookInvoicing and payments for freelancers and small businesses. Plans from $9 to $39 a month.
ceo_note.md1 lines · Download
From Dan (CEO): "Billfront is killing us. I want a full feature-by-feature comparison against them and a plan to match everything they have by Q2."
deals_last_two_quarters.csv (trials that became deals, won or lost)deal,segment,outcome,primary_reason,went_to T-900,Small business,won,, T-901,Freelancer,won,, T-902,Freelancer,won,, T-903,Small business,won,, T-904,Small business,lost,Accountant uses something else,Xero invoicing T-905,Small business,won,, T-906,Small business,lost,Didn't get round to switching,No decision T-907,Freelancer,won,, T-908,Freelancer,won,, T-909,Small business,won,, T-910,Freelancer,lost,Price,Billfront T-911,Freelancer,won,, T-912,Small business,lost,Price,Billfront T-913,Freelancer,won,, T-914,Freelancer,won,, T-915,Small business,won,, T-916,Freelancer,won,, T-917,Small business,won,, T-918,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-919,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-920,Freelancer,won,, T-921,Agency (multi-currency),won,, T-922,Small business,won,, T-923,Small business,won,, T-924,Small business,won,, T-925,Freelancer,won,, T-926,Freelancer,won,, T-927,Freelancer,won,, T-928,Small business,won,, T-929,Small business,lost,Didn't get round to switching,No decision T-930,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-931,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-932,Small business,won,, T-933,Small business,won,, T-934,Small business,won,, T-935,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-936,Agency (multi-currency),lost,Didn't get round to switching,No decision T-937,Small business,lost,Didn't get round to switching,No decision T-938,Agency (multi-currency),won,, T-939,Small business,lost,Didn't get round to switching,No decision T-940,Small business,won,, T-941,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-942,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-943,Freelancer,won,, T-944,Small business,won,, T-945,Small business,lost,Accountant uses something else,Xero invoicing T-946,Small business,won,, T-947,Small business,won,, T-948,Small business,won,, T-949,Small business,lost,Didn't get round to switching,No decision T-950,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-951,Freelancer,won,, T-952,Freelancer,won,, T-953,Freelancer,won,, T-954,Agency (multi-currency),won,, T-955,Small business,lost,Didn't get round to switching,No decision T-956,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-957,Freelancer,won,, T-958,Small business,won,, T-959,Freelancer,won,, T-960,Small business,lost,Didn't get round to switching,No decision T-961,Freelancer,won,, T-962,Small business,won,, T-963,Agency (multi-currency),won,, T-964,Small business,won,, T-965,Small business,won,, T-966,Small business,won,, T-967,Small business,lost,Didn't get round to switching,No decision T-968,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-969,Small business,won,, T-970,Freelancer,lost,Price,Billfront T-971,Small business,lost,Didn't get round to switching,No decision T-972,Small business,lost,Accountant uses something else,Xero invoicing T-973,Small business,lost,Accountant uses something else,Xero invoicing T-974,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-975,Small business,lost,Price,Billfront T-976,Freelancer,won,, T-977,Small business,lost,Didn't get round to switching,No decision T-978,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-979,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-980,Freelancer,won,, T-981,Small business,won,, T-982,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-983,Small business,won,, T-984,Small business,lost,Didn't get round to switching,No decision T-985,Freelancer,won,, T-986,Freelancer,won,, T-987,Small business,lost,Didn't get round to switching,No decision T-988,Small business,won,, T-989,Freelancer,won,, T-990,Freelancer,won,, T-991,Freelancer,won,, T-992,Agency (multi-currency),won,, T-993,Freelancer,won,, T-994,Freelancer,won,, T-995,Small business,won,, T-996,Small business,won,, T-997,Small business,won,, T-998,Small business,lost,Didn't get round to switching,No decision T-999,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-1000,Agency (multi-currency),won,, T-1001,Small business,lost,Price,Billfront T-1002,Small business,won,, T-1003,Freelancer,won,, T-1004,Agency (multi-currency),lost,Didn't get round to switching,No decision T-1005,Freelancer,won,, T-1006,Small business,won,, T-1007,Freelancer,won,, T-1008,Small business,won,, T-1009,Freelancer,lost,Price,Billfront T-1010,Small business,won,, T-1011,Freelancer,won,, T-1012,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-1013,Freelancer,won,, T-1014,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-1015,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-1016,Freelancer,won,, T-1017,Freelancer,won,, T-1018,Freelancer,won,, T-1019,Small business,lost,Didn't get round to switching,No decision T-1020,Freelancer,won,, T-1021,Small business,won,, T-1022,Agency (multi-currency),won,, T-1023,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-1024,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-1025,Freelancer,won,, T-1026,Freelancer,won,, T-1027,Freelancer,won,, T-1028,Small business,won,, T-1029,Freelancer,won,,
customers_by_segment.md7 lines · Download
# Tallybook customers and revenue (September)

| Segment | Paying customers | Average revenue per month |
|---|---|---|
| Freelancer | 5,800 | $11 |
| Small business | 3,100 | $24 |
| Agency (multi-currency) | 240 | $46 |
billfront_public_info.md6 lines · Download
# Billfront (public pages, September)

- Pricing: $12/month Starter, $29/month Pro (multi-currency, live FX rates), $59/month Team.
- Homepage: "Invoicing for global teams". Case studies: three agencies, one consultancy.
- Recent launches: multi-currency invoicing (March), client portal (June).
- Job posts: 4 engineers for "global payments", 2 for "agency workflows".
What a strong answer doesThe answer key the graders mark against

Shows from the deals file that the biggest competitor is no decision: of 46 lost deals, 25 ended with the prospect staying on spreadsheets or never switching, 17 went to Billfront and 4 to Xero invoicing. Billfront's wins are concentrated in multi-currency agencies (11 of its 17), on a specific gap (invoicing in several currencies with live rates); elsewhere it won only 6 deals, mostly on price. Agencies are a small segment (240 customers at about $46 a month, about $11,000 MRR, against about $64,000 from freelancers and $74,000 from small businesses), and Billfront is heading further into them (global payments and agency workflow hiring). Recommends against matching everything: fix switching friction for small businesses who 'didn't get round to it' (imports, setup help, accountant tie-ins), make a deliberate call on multi-currency (build it if agencies matter strategically, or cede the segment), and keep the advantages Billfront can't easily copy. Gives a short comparison only where it matters.

Critical failures (cap the score at 40)
  • Recommends matching Billfront feature for feature
Task

Mid-level · v1.0

The outputs

8 setups answered this brief.

Opus 5.5Claude98Usable with a quick editGPT-6 LunaAPI94Usable with a quick editGPT-6 AstraChatGPT94Usable with a quick editSonnet 5.5API94Usable with a quick editGPT-6.1 SolAPI94Usable with a quick editHaiku 5.5API92Usable with a quick editGemini 3.8 FlashAPI71Usable with a quick editGemini 3.5 Flash-LiteGemini35Needs more work
Gemini 3.8 FlashAPI

Usable with a quick edit6 of 11 checks passed

The output is well-structured, evidence is correctly used, and the recommendation is clear. A PM could use it with minor wording edits (e.g., softening the 'vacating' claim) without re-checking data or redoing reasoning.

Output

MEMORANDUM

TO: Dan Whitlock, CEO FROM: PM for Growth DATE: October 14, 2023 SUBJECT: Competitive Analysis: Billfront & Win/Loss Findings

---

Executive Summary

Billfront is not killing us. Across the last two quarters (130 evaluated deals), our win rate stands at 65% (84 wins).

Of the 46 deals lost, 54% (25 deals) were lost to inertia and the status quo (“No decision”), not a competitor. Billfront won 17 deals (37% of losses), but 65% of those losses were concentrated in a single niche: cross-border agencies needing live multi-currency rates.

Matching Billfront feature-for-feature by Q2 would misallocate our roadmap to chase a customer segment that accounts for just 7.4% of our revenue, while ignoring the friction costing us half of our lost pipeline in our core market.

---

1. Who We Really Lose To

Our win/loss data over the last two quarters paints a very clear picture of why prospective customers fail to convert:

``` Loss Reasons Across All Segments (46 Lost Deals) ┌───────────────────────────────────────────────┬───────┬───────┐ │ Competitor / Outcome │ Deals │ Share │ ├───────────────────────────────────────────────┼───────┼───────┤ │ No Decision: "Didn't get round to switching" │ 16 │ 35% │ │ Billfront: Multi-currency FX gap │ 11 │ 24% │ │ No Decision: Stayed on Spreadsheets / Word │ 9 │ 20% │ │ Billfront: Price │ 6 │ 13% │ │ Xero: Accountant preference │ 4 │ 9% │ └───────────────────────────────────────────────┴───────┴───────┘ ```

  • #1 Competitor: Status Quo & Inertia (25 deals / 54% of losses). Small businesses abandon onboarding because switching is too painful (14 deals), and freelancers default back to Word/Excel templates (9 deals).
  • #2 Competitor: Billfront (17 deals / 37% of losses). 11 of these were multi-currency agencies; only 6 were core domestic freelancers or small businesses citing price.
  • #3 Competitor: Accounting Ecosystem Lock-in (4 deals / 9% of losses). Small businesses whose external accountants mandate Xero.

---

2. Where Billfront Is (and Isn't) a Threat

To evaluate Billfront’s threat level, we must cross-reference our losses against where our revenue actually comes from:

Customer SegmentPaying CustomersARPUMonthly Revenue% of Total MRR
:---:---:---:---:---
Freelancer5,800$11$63,80042.7%
Small Business3,100$24$74,40049.9%
Agency (Multi-Currency)240$46$11,0407.4%
Total9,140—$149,240100%

#### Where Billfront is a real threat: * Multi-Currency Agencies: Billfront launched multi-currency invoicing with live FX in March and is winning on that capability. We lost 11 agency deals specifically on this feature. If an agency invoices international clients in varying currencies, we currently do not compete.

#### Where Billfront is NOT a threat: * Freelancers and Domestic SMBs: These two groups generate 92.6% of our revenue ($138k MRR). Out of 110 pipeline trials in these segments, Billfront took only 6 deals (5.5%). * Billfront’s starting tier is $12/month (vs. our $9/month). The 6 price losses suggest confusion around feature gating between our tiers rather than Billfront undercutting our baseline price.

---

3. Where Billfront Is Heading

Billfront’s public positioning and hiring roadmap reveal an explicit upmarket strategy: * Positioning: Rebranded to "Invoicing for global teams", with agency-only case studies. * Recent Releases: Client portal (June), multi-currency (March). * Hiring: Actively recruiting 4 engineers for "global payments" and 2 for "agency workflows".

Billfront is vacating the simple domestic SMB and freelancer market to become specialized back-office software for distributed, cross-border agencies. Chasing them feature-for-feature means chasing them into an enterprise agency tier.

---

4. Strategic Recommendation: What We Should Do Instead

Matching Billfront’s entire product by Q2 would consume all engineering resources on agency workflows (portals, complex multi-currency ledgering, global payments) for our smallest segment, while leaving our 93% revenue engine vulnerable to churn and onboarding friction.

Instead, I propose a focused, dual-track plan for Q1–Q2:

#### Pillar 1: Attack the 54% "No Decision" Leak (Growth Priority) Reducing activation friction will yield 3x the revenue impact of chasing agency features: 1. Frictionless Migration Tooling: Build 1-click CSV/spreadsheet import for customer lists and open invoices to eliminate the "didn't get round to switching" barrier for SMBs. 2. Template-to-App Onboarding: Create lightweight web invoice generators that let spreadsheet users experience Tallybook’s value before forcing account creation. 3. Accountant Collaboration View: Introduce a basic accountant export/read-only seat to stop leakage to Xero.

#### Pillar 2: Selective Multi-Currency Response (Monetization Priority) We should not build custom "agency workflows," but we should neutralize the specific product gap causing deal losses: 1. Targeted Multi-Currency & Live FX: Integrate automated FX exchange rates for cross-border invoicing by Q2. 2. Package as a Premium Tier / Add-on: Billfront charges $29/mo (Pro) and $59/mo (Team) for this. We can package multi-currency into a dedicated $49/mo "International Tier" or a paid add-on, immediately lifting agency ARPU without bloating the product for domestic users.

#### Pillar 3: Audit Pricing Tiers Investigate the 6 lost deals to Billfront on price. Ensure our $9 entry plan is clearly differentiated and does not wall off basic capabilities that drive price-sensitive users away.

---

Next Steps

I recommend holding off on an across-the-board feature-parity initiative. I will deliver a detailed spec and ROI projection by Friday for (1) Migration/Switching Flow Optimization and (2) Multi-Currency Invoicing as a Premium Add-on for Q1 delivery.

Claims the judge couldn’t find in the brief

  • Billfront is vacating the simple domestic SMB and freelancer market to become specialized back-office software for distributed, cross-border agencies.

Grades and run details

Decision model 73 · LLM judge 67
Decision model checks
  • failUses the supplied evidence correctly20%
  • passAddresses the actual decision41%
  • passRespects explicit constraints35%
  • partialIdentifies material uncertainty46%
  • failAvoids unsupported claims25%
  • passProduces the required deliverable70%
  • passFinds the real competitor96%
  • passPins down where Billfront wins100%
  • passCompetes with what customers really choose100%
  • passGets the base of every number right13%
  • partialSeparates what a rival can copy65%
Run
Run
#1
API response time
26 s
Submitted
6 Oct 2026
Sonnet 5.5API

Usable with a quick edit10 of 11 checks passed

The analysis is clear, data-backed, and actionable; a PM could use it with minor wording edits without rechecking evidence or redoing reasoning.

Output

To: Dan Whitlock From: Growth PM Re: Billfront: what the deal data says

Bottom line

Billfront is a real threat in one segment, not across the board. Most of our losses aren't to Billfront at all. I recommend we don't match everything by Q2. We should build the one feature that is costing us deals, and fix the larger problem, which is trials that never convert.

Who we lose to

In the last two quarters we closed 84 of 130 trials and lost 46:

Lost toDealsShare of losses
No decision (didn't switch, or stayed on spreadsheets/Word)2554%
Billfront1737%
Xero invoicing (accountant's choice)49%

Billfront took 17 of 130 trials, about 13%. Most of our losses are people who never chose anyone.

Where Billfront is a threat

Agencies needing multi-currency invoicing with live rates. Of 20 agency trials, we won 7 (35%). We lost 11 to Billfront on this one capability, and every one of them gave exactly that reason. Billfront launched it in March and bundles it in its $29 Pro plan. Our agency customers average $46 a month.

Outside agencies, Billfront barely registers. Win rates are 77% for freelancers (41 of 53) and 63% for small businesses (36 of 57). Billfront beat us on price in 6 deals, 3 freelancer and 3 small business, which is about 5% of trials in each segment.

The price losses are odd. Billfront's Starter is $12 and ours starts at $9, so list price at the entry tier doesn't explain them. I don't know what those prospects compared. We should find out before discussing any price changes.

Sizing it in revenue

Using segment average revenue (assuming lost deals would have paid the average):

  • Agency multi-currency losses to Billfront: 11 × $46 ≈ $506/month
  • No-decision losses: 14 small business × $24, 9 freelancer × $11, 2 agency × $46 ≈ $527/month
  • Billfront price losses: ≈ $105/month
  • Xero losses: ≈ $96/month

The multi-currency gap and the no-decision problem are about the same size. The multi-currency losses come from 11 deals, while the no-decision losses are spread across 25 and are probably cheaper to fix.

Agencies are only 240 of our 9,140 paying customers and about 7% of monthly revenue ($11k of $149k). They are our highest-value segment per customer, though, and they are where Billfront is heading.

Where Billfront is heading

Every public signal points at agencies and global teams, not at our core:

  • The homepage says "Invoicing for global teams".
  • Its case studies are three agencies and one consultancy.
  • It launched multi-currency in March and a client portal in June.
  • Of its open engineering roles, 4 are for "global payments" and 2 for "agency workflows".
  • It has a $59 Team tier above anything we sell.

A full-parity plan means following a roadmap aimed at a segment that is 7% of our revenue. It also means matching features like the client portal, which appears in none of our loss reasons. The same data suggests it's a risk to our 240 existing agency customers, not just to new deals. That's about $11k a month to defend.

What I recommend

  1. Build multi-currency invoicing with live FX rates, scoped tightly, as our one parity feature. It's the only feature with direct evidence behind it (11 lost deals). I need an engineering estimate before committing to Q2. I'd also check whether our current agency customers are asking for it, since that's the retention case.
  1. Attack "didn't get round to switching" and spreadsheet users. This is 25 deals and 54% of losses, concentrated in small business (14) and freelancers (9). Candidate fixes are guided import of existing clients and invoices, template migration from Word and spreadsheets, and a trial-end nudge with a done-for-you setup offer. We don't yet know why these people stalled. Before building, we should run about 10 calls with them. That costs days, not a quarter.
  1. Look into the Xero losses, but keep it small. All four came from accountants preferring another tool. A light accountant referral or integration push is worth a look, but it's 9% of losses and I wouldn't prioritise it above the other items.
  1. Investigate the price losses. I'd ask lost prospects what plan comparison they made. I wouldn't change pricing on 6 deals.
  1. Monitor Billfront rather than chase it. I'll track its releases and pricing quarterly and re-check agency win rate. If its pitch starts working on small businesses with teams, we revisit.

Caveats

  • The sample is 130 trials, and the loss reasons are as recorded by the team, not verified with prospects.
  • The data has no dates, so I can't tell whether the agency losses accelerated after Billfront's March launch. I'd add dates to the loss data.
  • I only have Billfront's public pages, so I can't do a true feature-by-feature comparison. I can build one from their product and pricing pages if you still want it, but I'd treat it as reference, not as a roadmap.

I'm glad to walk through the numbers, and I'll bring the engineering estimate and call findings to our next review.

Grades and run details

Decision model 95 · LLM judge 92
Decision model checks
  • passUses the supplied evidence correctly40%
  • passAddresses the actual decision76%
  • passRespects explicit constraints62%
  • passIdentifies material uncertainty97%
  • passAvoids unsupported claims42%
  • passProduces the required deliverable46%
  • passFinds the real competitor100%
  • passPins down where Billfront wins100%
  • passCompetes with what customers really choose100%
  • passGets the base of every number right35%
  • partialSeparates what a rival can copy62%
Run
Run
#1
API response time
43 s
Submitted
6 Oct 2026
Gemini 3.5 Flash-LiteGemini

Needs more than a quick edit2 of 11 checks passed

The core data analysis is wrong, making the entire competitive assessment unreliable. A PM would need to re-analyze the deals file from scratch.

Output

MEMORANDUM

TO: Dan Whitlock, CEO FROM: Growth PM DATE: October 24, 2023 SUBJECT: Competitive Analysis: Tallybook vs. Billfront & Q2 Growth Strategy

---

Dan,

You noted that "Billfront is killing us" and asked for a feature match by Q2. I’ve dug into our last two quarters of deal data (trials T-900 through T-1029) and analyzed Billfront’s public footprint.

The data tells a surprising story: Billfront is not killing us broadly. In fact, we are winning the vast majority of our trials. However, Billfront is systematically poaching a very specific, high-value segment.

Below is our competitive reality check, a feature comparison, and a smarter Q2 plan than simply copying their roadmap.

---

1. Who We Really Lose To (The Data)

Out of 130 trial outcomes analyzed from the last two quarters, 74 trials were won (57%) and 56 were lost (43%).

When we lose, where do those deals actually go? * No Decision / Inertia (38 lost / 68% of losses): * Stayed on spreadsheets / Word templates: 16 losses. * Didn't get round to switching: 22 losses. * Takeaway: Our biggest enemy isn't a competitor; it's friction and customer apathy. * Xero (4 losses / 7% of losses): Small businesses whose accountants forced them onto an enterprise ledger. * Billfront (14 losses / 25% of losses): * Price: 4 freelancer losses (lost to their $12 entry tier vs our $9–$11 range). * Needs invoicing in several currencies with live rates: 10 agency losses.

The Verdict: Billfront is only a threat in the Multi-Currency Agency segment. They are positioning themselves as a vertical tool for global agencies, while we are trying to be a generalist tool.

---

2. Feature-by-Feature Comparison: Tallybook vs. Billfront

Feature / CapabilityTallybook ($9–$39/mo)Billfront ($12–$59/mo)Where Billfront is a Real ThreatWhere It’s Noise
:---:---:---:---:---
Core InvoicingRobust, clean templatesStandard invoicingNeutralBoth handle basic billing effortlessly.
Pricing & Packaging$9 (Freelancer), $24 (SMB), $46 (Agency ARPU)$12 (Starter), $29 (Pro), $59 (Team)Real Threat: Their $12 tier undercuts our SMB/Freelancer fence for price-sensitive buyers.
Multi-Currency & Live FXBasic manual entry / limited supportNative live FX rates & multi-currencyLethal Threat: 100% of our agency losses cite this exact feature.
Client PortalBasic payment link viewAdvanced portal (invoices, history, downloads)MinorNice-to-have, but didn't cause any documented churn/loss.
Global Payments / TeamsSingle-user focus, standard gatewaysLocalization, team workflows, agency toolsFuture Threat: Their current hiring (4 engineers for global payments, 2 for agency workflows) proves they are doubling down here.

---

3. Where Billfront is Heading

Based on their job postings and recent launches (client portal in June, multi-currency in March), Billfront is moving upmarket into localized, multi-entity agency operations. They are deliberately abandoning the low-end freelancer market to capture high-value agencies paying $59+/month.

If we spend Q2 blindly matching their features (like building a complex client portal or team permissions), we will burn engineering cycles copying a product that is deliberately scaling away from our bread-and-butter.

---

4. Strategic Recommendation: What We Should Do Instead of Matching

Our revenue is anchored by 5,800 Freelancers ($11 ARPU) and 3,100 Small Businesses ($24 ARPU). Our 240 Agencies ($46 ARPU) are nice, but they represent a fraction of our base.

Instead of playing catch-up to Billfront in a market we aren't optimized to win, we should execute a three-part growth plan for Q2:

#### A. Surrender the Multi-Currency Agency Segment (Deliberately) Stop trying to win agencies that demand live FX rates. Building a multi-currency engine requires heavy compliance, banking integrations, and engineering overhead to chase 10 lost deals a quarter. Let Billfront have them.

#### B. Fix the Real Leaks: Inertia and Onboarding (High ROI) Since 68% of our losses are to "No Decision," our biggest growth lever is reducing time-to-first-invoice for people stuck on spreadsheets. * Action: Build a 1-click CSV importer for past invoices and client lists in Q2. If we can convert half of the "spreadsheet/word template" drop-offs, we instantly capture 8+ new accounts a quarter without touching our pricing.

#### C. Protect the SMB Flank with Smart Packaging Billfront is picking off price-sensitive freelancers with their $12 tier. * Action: Introduce a streamlined "Solo" tier at $10/mo with zero friction, and double down on Tallybook's core differentiator: speed and simplicity for domestic small businesses and freelancers who don't care about global FX rates, they just want to get paid fast.

Summary: Billfront isn't killing us—they are just taking the multi-currency headaches off our hands. Let’s focus Q2 engineering power on wiping out spreadsheet inertia rather than chasing Billfront's agency ghost.

Best, Growth PM, Tallybook

Claims the judge couldn’t find in the brief

  • Out of 130 trial outcomes, 74 trials were won (57%) and 56 were lost (43%).
  • No Decision / Inertia accounts for 38 lost deals (68% of losses): 16 stayed on spreadsheets, 22 didn't get round to switching.
  • Billfront took 14 losses (25% of losses): 4 on price, 10 on multi-currency.
  • Tallybook's core invoicing has robust, clean templates.
  • Tallybook's multi-currency support is basic manual entry / limited support.
  • Tallybook's client portal is a basic payment link view.
  • Tallybook has a single-user focus and standard payment gateways.
  • Building a multi-currency engine requires heavy compliance, banking integrations, and engineering overhead.
  • Billfront is deliberately abandoning the low-end freelancer market.
  • Billfront's $12 tier undercuts our SMB/Freelancer fence for price-sensitive buyers.

Grades and run details

Decision model 59 · LLM judge 17
Decision model checks
  • failUses the supplied evidence correctly51%
  • passAddresses the actual decision32%
  • passRespects explicit constraints12%
  • failIdentifies material uncertainty73%
  • failAvoids unsupported claims55%
  • passProduces the required deliverable56%
  • passFinds the real competitor97%
  • passPins down where Billfront wins59%
  • passCompetes with what customers really choose99%
  • failGets the base of every number right59%
  • partialSeparates what a rival can copy67%
Run
Run
#1
Time to output
7 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 8

Uses the supplied evidence correctlyMixedRightWrong
Gemini 3.8 Flash · API

All factual claims about the current situation are taken correctly from the supplied context or derived by accurate arithmetic; the one unsupported claim is a forecast, not a current-situation fact.

Sonnet 5.5 · API

All factual claims about the current situation are directly supported by the supplied context or arithmetic.

Gemini 3.5 Flash-Lite · Gemini

Key numbers (wins, losses, breakdown by reason) are incorrect, and many feature claims about Tallybook are invented.

Addresses the actual decisionMixedRightMixed
Gemini 3.8 Flash · API

The output commits to a clear recommendation (don't match everything, do Pillars 1-3) but does not state what result or condition would change that call, as required.

Sonnet 5.5 · API

Commits to a clear recommendation (don't match everything, build multi-currency, fix no-decision) and states what would change it (Billfront pitch working on small businesses).

Gemini 3.5 Flash-Lite · Gemini

Does not state what result or condition would change the recommendation.

Identifies material uncertaintyWrongRightWrong
Gemini 3.8 Flash · API

The output does not name specific unknowns that could change the decision, nor say how they would be resolved or what would change the call.

Sonnet 5.5 · API

Names unknowns (price comparison details, reasons for stalling, engineering estimate, agency customer demand, missing dates) and how to resolve them.

Gemini 3.5 Flash-Lite · Gemini

No specific unknowns that could change the decision are named, nor how they would be resolved.

Avoids unsupported claimsWrongRightWrong
Gemini 3.8 Flash · API

The claim that 'Billfront is vacating the simple domestic SMB and freelancer market' is presented as an established fact, but the evidence only shows they are targeting agencies, not that they are leaving other markets.

Sonnet 5.5 · API

Interpretations and forecasts are labelled as assumptions or hypotheses (e.g., 'probably cheaper to fix', 'I don't know what those prospects compared').

Gemini 3.5 Flash-Lite · Gemini

Presents interpretations (e.g., 'deliberately abandoning the low-end market') and feature details as fact without support.

Produces the required deliverableRightRightMixed
Gemini 3.8 Flash · API

The memo is complete, within 900 words, addressed to Dan, and provides a usable competitive analysis with a clear recommendation.

Sonnet 5.5 · API

Delivers a complete, actionable competitive analysis in memo form for the CEO, within the word limit.

Gemini 3.5 Flash-Lite · Gemini

The analysis contains wrong numbers, so a PM cannot act on it without redoing the data work.

Finds the real competitorRightRightMixed
Gemini 3.8 Flash · API

The analysis counts losses by destination, shows 25 no-decision losses vs 17 to Billfront, and names no decision as the biggest competitor.

Sonnet 5.5 · API

Counts losses by destination and names no decision as the biggest (25 deals, 54% of losses).

Gemini 3.5 Flash-Lite · Gemini

Incorrectly counts no-decision losses as 38 instead of 25, and misstates the percentage.

Pins down where Billfront winsRightRightMixed
Gemini 3.8 Flash · API

It shows Billfront's wins are concentrated in multi-currency agencies (11 of 17) and sizes the agency segment at 7.4% of MRR against freelancer and small business.

Sonnet 5.5 · API

Shows Billfront's wins are concentrated in multi-currency agencies (11 of 17) and sizes the segment's revenue ($11k of $149k).

Gemini 3.5 Flash-Lite · Gemini

Miscounts Billfront losses and misstates the concentration; does not correctly size the segment's revenue against others with accurate numbers.

Gets the base of every number rightRightRightWrong
Gemini 3.8 Flash · API

All derived percentages and rates use the correct denominators (130 deals, 46 losses, 110 freelancer+small business trials) and check out against the data.

Sonnet 5.5 · API

All percentages and rates are computed correctly from the right denominators, with bases stated where needed.

Gemini 3.5 Flash-Lite · Gemini

All derived percentages and counts are based on wrong win/loss totals and wrong sub-counts.

All got wrong 1

Separates what a rival can copyWrongWrongWrong
Gemini 3.8 Flash · API

The output does not explicitly name which advantages are copyable and which are durable, nor anchor the response in the latter, as required.

Sonnet 5.5 · API

Does not explicitly separate advantages a competitor could copy (features, price) from durable ones (switching costs, relationships) or anchor the response in the latter.

Gemini 3.5 Flash-Lite · Gemini

Does not separate copyable advantages from durable ones; mentions 'speed and simplicity' but doesn't analyze its defensibility.

All got right 2

Respects explicit constraintsRightRightRight
Gemini 3.8 Flash · API

The output is a competitive analysis memo under 900 words, addressed to the CEO, and does not propose a full feature-by-feature matching plan, respecting the brief's request to say what to do instead.

Sonnet 5.5 · API

The output is a competitive analysis memo to Dan, under 900 words, and does not propose a full feature-by-feature matching plan.

Gemini 3.5 Flash-Lite · Gemini

The output is a memo under 900 words, addresses the CEO, and does not propose a feature-by-feature matching plan.

Competes with what customers really chooseRightRightRight
Gemini 3.8 Flash · API

The analysis treats spreadsheets and inertia as a competitor, and draws a direction-of-travel insight (Billfront moving upmarket) that affects the recommendation.

Sonnet 5.5 · API

Treats spreadsheets/manual work as a competitor and draws direction-of-travel insight from Billfront's hiring and launches.

Gemini 3.5 Flash-Lite · Gemini

Treats spreadsheets and inertia as the main competitor and draws a direction-of-travel insight from Billfront's hiring.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Opus 5.5Claude86.4100.02None
2GPT-6 AstraChatGPT88.695.82None
3GPT-6.1 SolAPI88.695.82None
4GPT-6 LunaAPI93.283.32None
5Sonnet 5.5API86.483.32None
6Haiku 5.5API84.179.22None
7Gemini 3.8 FlashAPI68.254.22None
8Gemini 3.5 Flash-LiteGemini61.425.02None

What you get for the cost

GPT-6.1 Sol · API scores within a point of Opus 5.5 · Claude at a fifth of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreOpus 5.5 · Claude: task score 94, 75× the cheapestGPT-6 Astra · ChatGPT: task score 93, 87× the cheapestGPT-6.1 Sol · API: task score 93, 14× the cheapestGPT-6 Luna · API: task score 86, the cheapestSonnet 5.5 · API: task score 83, 44× the cheapestHaiku 5.5 · API: task score 80, 3.0× the cheapestGemini 3.8 Flash · API: task score 60, 12× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 41, 2.9× the cheapestOpus 5.5 · ClaudeGPT-6 Astra · ChatGPTGPT-6.1 Sol · APIGPT-6 Luna · APISonnet 5.5 · APIHaiku 5.5 · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and cost

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Deciding how to respond to competitors.

Why it matters

Feature grids hide the real competition, which is often the customer's current workaround, and the response that feels urgent is usually to copy everything.

What good looks like

  • Includes the status quo and non-obvious alternatives
  • Finds where losses really go, by segment
  • Separates advantages a rival can copy from ones it can't
  • Reads where competitors are heading

Deliberately not measured

  • Market sizing
Capability tested

Competitive insight

The failure we’re looking for

A feature checklist and a plan to match everything

Grading

Decision model and LLM judge, calibrated against a blind PM review