Humanity’s Last PRD
Latest drop: Gemini 4 Argon · 13 Oct
Tasks / Strategy

Competitor analysis

Can the model find where a competitor can actually hurt you, from win/loss and churn data, rather than list features?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 56% were usable with at most a quick edit.

Reliably right

  1. Respects explicit constraints100% pass
    Delivers a competitive analysis memo to Dan within 900 words, addressing all requested topics without proposing a full feature-by-feature match.
    Opus 5.5 · Claude · The competitor isn't who you think
  2. Reads where Crestline is heading100% pass
    It reads Crestline's job ads as signalling payroll, mid-market, and possible accountant partnerships, and adjusts the response accordingly.
    GPT-6.1 Sol · API · Free forever, funded
  3. Produces the required deliverable97% pass
    The memo is complete, in the right form for the CEO, within the word limit, and actionable with light edits.
    Opus 5.5 · Claude · The competitor isn't who you think

Where it slips

  1. Uses the supplied evidence correctly56% pass
    Two factual claims are wrong: the number of unconnected multi-user accounts going to Crestline is 40, not 42, and the number of connected 6-20 cancellations is 5, not 4.
    Sonnet 5.5 · API · Free forever, funded
  2. Gets the base of every number right63% pass
    Derived figures for unconnected multi-user cancellations (42 vs actual 40) and connected 6-20 cancellations (4 vs actual 5) are incorrect.
    Sonnet 5.5 · API · Free forever, funded
  3. Separates what a rival can copy68% pass
    Does not explicitly separate advantages a well-funded competitor could copy from those it couldn't, nor anchor the response in durable advantages with evidence.
    GPT-6 Luna · API · The competitor isn't who you think

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Competes with what customers really choose

    Does the analysis treat the status quo (spreadsheets, manual work, doing nothing) as a competitor, and draw at least one implication from where competitors are investing rather than only from today's features?

    Passes when Analyses the status quo, strengths included, and gives a direction-of-travel insight that affects the recommendation.

    April DunfordLenny’s Podcast

  2. Separates what a rival can copy

    Does the analysis separate advantages a well-funded competitor could copy (features, price, being first) from ones it couldn't easily (switching costs, networks, relationships), and build the response on the latter?

    Passes when Names which advantages are copyable and which aren't, with evidence, and anchors the response in the durable ones.

    Hamilton HelmerLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for growth at Tallybook. Our CEO, Dan Whitlock, wants a feature-by-feature comparison against Billfront and a plan to match everything they have by Q2 (his note is below). Write him a competitive analysis of no more than 900 words: who we really lose to, where Billfront is a real threat and where it isn't, where it's heading, and what we should do instead of (or as well as) matching it. The files are attached.

What the model was given5 items: About Tallybook, ceo_note.md, deals_last_two_quarters.csv (trials that became deals, won or lost), customers_by_segment.md, billfront_public_info.md
About TallybookInvoicing and payments for freelancers and small businesses. Plans from $9 to $39 a month.
ceo_note.md1 lines · Download
From Dan (CEO): "Billfront is killing us. I want a full feature-by-feature comparison against them and a plan to match everything they have by Q2."
deals_last_two_quarters.csv (trials that became deals, won or lost)deal,segment,outcome,primary_reason,went_to T-900,Small business,won,, T-901,Freelancer,won,, T-902,Freelancer,won,, T-903,Small business,won,, T-904,Small business,lost,Accountant uses something else,Xero invoicing T-905,Small business,won,, T-906,Small business,lost,Didn't get round to switching,No decision T-907,Freelancer,won,, T-908,Freelancer,won,, T-909,Small business,won,, T-910,Freelancer,lost,Price,Billfront T-911,Freelancer,won,, T-912,Small business,lost,Price,Billfront T-913,Freelancer,won,, T-914,Freelancer,won,, T-915,Small business,won,, T-916,Freelancer,won,, T-917,Small business,won,, T-918,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-919,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-920,Freelancer,won,, T-921,Agency (multi-currency),won,, T-922,Small business,won,, T-923,Small business,won,, T-924,Small business,won,, T-925,Freelancer,won,, T-926,Freelancer,won,, T-927,Freelancer,won,, T-928,Small business,won,, T-929,Small business,lost,Didn't get round to switching,No decision T-930,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-931,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-932,Small business,won,, T-933,Small business,won,, T-934,Small business,won,, T-935,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-936,Agency (multi-currency),lost,Didn't get round to switching,No decision T-937,Small business,lost,Didn't get round to switching,No decision T-938,Agency (multi-currency),won,, T-939,Small business,lost,Didn't get round to switching,No decision T-940,Small business,won,, T-941,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-942,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-943,Freelancer,won,, T-944,Small business,won,, T-945,Small business,lost,Accountant uses something else,Xero invoicing T-946,Small business,won,, T-947,Small business,won,, T-948,Small business,won,, T-949,Small business,lost,Didn't get round to switching,No decision T-950,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-951,Freelancer,won,, T-952,Freelancer,won,, T-953,Freelancer,won,, T-954,Agency (multi-currency),won,, T-955,Small business,lost,Didn't get round to switching,No decision T-956,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-957,Freelancer,won,, T-958,Small business,won,, T-959,Freelancer,won,, T-960,Small business,lost,Didn't get round to switching,No decision T-961,Freelancer,won,, T-962,Small business,won,, T-963,Agency (multi-currency),won,, T-964,Small business,won,, T-965,Small business,won,, T-966,Small business,won,, T-967,Small business,lost,Didn't get round to switching,No decision T-968,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-969,Small business,won,, T-970,Freelancer,lost,Price,Billfront T-971,Small business,lost,Didn't get round to switching,No decision T-972,Small business,lost,Accountant uses something else,Xero invoicing T-973,Small business,lost,Accountant uses something else,Xero invoicing T-974,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-975,Small business,lost,Price,Billfront T-976,Freelancer,won,, T-977,Small business,lost,Didn't get round to switching,No decision T-978,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-979,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-980,Freelancer,won,, T-981,Small business,won,, T-982,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-983,Small business,won,, T-984,Small business,lost,Didn't get round to switching,No decision T-985,Freelancer,won,, T-986,Freelancer,won,, T-987,Small business,lost,Didn't get round to switching,No decision T-988,Small business,won,, T-989,Freelancer,won,, T-990,Freelancer,won,, T-991,Freelancer,won,, T-992,Agency (multi-currency),won,, T-993,Freelancer,won,, T-994,Freelancer,won,, T-995,Small business,won,, T-996,Small business,won,, T-997,Small business,won,, T-998,Small business,lost,Didn't get round to switching,No decision T-999,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-1000,Agency (multi-currency),won,, T-1001,Small business,lost,Price,Billfront T-1002,Small business,won,, T-1003,Freelancer,won,, T-1004,Agency (multi-currency),lost,Didn't get round to switching,No decision T-1005,Freelancer,won,, T-1006,Small business,won,, T-1007,Freelancer,won,, T-1008,Small business,won,, T-1009,Freelancer,lost,Price,Billfront T-1010,Small business,won,, T-1011,Freelancer,won,, T-1012,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-1013,Freelancer,won,, T-1014,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-1015,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-1016,Freelancer,won,, T-1017,Freelancer,won,, T-1018,Freelancer,won,, T-1019,Small business,lost,Didn't get round to switching,No decision T-1020,Freelancer,won,, T-1021,Small business,won,, T-1022,Agency (multi-currency),won,, T-1023,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-1024,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-1025,Freelancer,won,, T-1026,Freelancer,won,, T-1027,Freelancer,won,, T-1028,Small business,won,, T-1029,Freelancer,won,,
customers_by_segment.md7 lines · Download
# Tallybook customers and revenue (September)

| Segment | Paying customers | Average revenue per month |
|---|---|---|
| Freelancer | 5,800 | $11 |
| Small business | 3,100 | $24 |
| Agency (multi-currency) | 240 | $46 |
billfront_public_info.md6 lines · Download
# Billfront (public pages, September)

- Pricing: $12/month Starter, $29/month Pro (multi-currency, live FX rates), $59/month Team.
- Homepage: "Invoicing for global teams". Case studies: three agencies, one consultancy.
- Recent launches: multi-currency invoicing (March), client portal (June).
- Job posts: 4 engineers for "global payments", 2 for "agency workflows".
What a strong answer doesThe answer key the graders mark against

Shows from the deals file that the biggest competitor is no decision: of 46 lost deals, 25 ended with the prospect staying on spreadsheets or never switching, 17 went to Billfront and 4 to Xero invoicing. Billfront's wins are concentrated in multi-currency agencies (11 of its 17), on a specific gap (invoicing in several currencies with live rates); elsewhere it won only 6 deals, mostly on price. Agencies are a small segment (240 customers at about $46 a month, about $11,000 MRR, against about $64,000 from freelancers and $74,000 from small businesses), and Billfront is heading further into them (global payments and agency workflow hiring). Recommends against matching everything: fix switching friction for small businesses who 'didn't get round to it' (imports, setup help, accountant tie-ins), make a deliberate call on multi-currency (build it if agencies matter strategically, or cede the segment), and keep the advantages Billfront can't easily copy. Gives a short comparison only where it matters.

Critical failures (cap the score at 40)
  • Recommends matching Billfront feature for feature
Task

Mid-level · v1.2

The outputs

9 setups answered this brief.

Opus 5.5Claude98Usable with a quick editGPT-6 LunaAPI94Usable with a quick editGPT-6 AstraChatGPT94Usable with a quick editSonnet 5.5API94Usable with a quick editGPT-6.1 SolAPI94Usable with a quick editHaiku 5.5API92Usable with a quick editGemini 3.8 FlashAPI73Usable with a quick editGemini 4 ArgonAPI52Needs more workGemini 3.5 Flash-LiteGemini31Needs more work
Gemini 4 ArgonAPI

Needs more than a quick edit5 of 11 checks passed

The output contains multiple numerical errors (Billfront loss count, no-decision count, trial count) that would require re-checking the data and redoing parts of the analysis. It cannot be used with just a quick edit.

Output

To: Dan Whitlock, CEO From: PM, Growth Subject: Billfront Competitive Analysis & Q2 Growth Plan

Dan,

Before building a Q2 roadmap to match Billfront feature-for-feature, I pulled our win/loss data from the last two quarters (130 trials) alongside our revenue breakdown and Billfront’s recent activity.

The data shows that Billfront is not killing us in our core business. Matching their roadmap by Q2 would pull engineering away from our biggest revenue drivers to chase a small, non-core segment. Here is what is actually happening in our pipeline and what we should do instead.

---

1. Who We Really Lose To: Inertia, Not Billfront

Out of 130 trials over the last two quarters, we won 84 (65%) and lost 46. When we lose, it usually isn’t to a software competitor—it’s to status quo inertia:

  • No Decision / Inertia (26 losses — 57% of all losses):
  • 17 deals (15 Small Business, 2 Agency) simply "didn't get round to switching."
  • 9 deals (all Freelancers) "stayed on spreadsheets or Word templates."
  • Billfront (16 losses — 35% of losses, 12% of total trials):
  • 10 deals were Multi-Currency Agencies needing live FX rates.
  • 6 deals (3 Freelancer, 3 Small Business) cited price.
  • Xero Invoicing (4 losses — 9% of losses):
  • All 4 were Small Businesses whose "accountant uses something else."

The takeaway: We lose 63% more deals to friction during onboarding and switching than we do to Billfront.

---

2. Where Billfront Is a Real Threat—And Where It Isn't

Where they are a threat: Multi-Currency Agencies In the Agency segment, Billfront is beating us. Out of 19 Agency trials, we won 7 and lost 12—10 of which went directly to Billfront because they launched multi-currency invoicing with live FX rates in March.

However, we need to put this segment in perspective against our current monthly recurring revenue ($149.2k total MRR): * Freelancers: 5,800 customers @ $11 ARPU = $63,800/mo (43% of MRR) * Small Businesses: 3,100 customers @ $24 ARPU = $74,400/mo (50% of MRR) * Agencies (multi-currency): 240 customers @ $46 ARPU = $11,040/mo (7% of MRR)

Where they are NOT a threat: Our Core (93% of our revenue) Across Freelancers and Small Businesses (111 combined trials), we lost only 6 deals to Billfront—all on "price," not missing features.

Notably, our plans ($9–$39/mo) are actually cheaper at the entry and top tiers than Billfront’s ($12 Starter, $29 Pro, $59 Team). Losing 6 core deals on "price" indicates a tier-packaging friction—specifically, small businesses or freelancers hitting a paywall forcing them onto our $39 plan when Billfront’s $29 Pro tier covers their needs—not a product gap.

---

3. Where Billfront Is Heading

Billfront is pivoting upmarket to become a workflow and cross-border payments platform for agencies and consultancies: * Positioning: Their homepage now leads with "Invoicing for global teams," backed exclusively by agency and consultancy case studies. * Recent Launches: Multi-currency invoicing (March) and a client portal (June). * Hiring: They are recruiting 4 engineers for "global payments" and 2 for "agency workflows." * Pricing: Their new $59/month Team tier reflects an upmarket push.

If we try to match everything they have by Q2 (client portals, agency workflows, global payments infrastructure), we will spend the next two quarters building complex software for 7% of our customer base while neglecting the 93% who just want fast, simple invoicing.

---

4. What We Should Do Instead by Q2

Rather than a feature-for-feature clone of Billfront, I recommend a focused three-part plan for Q2:

#### A. Attack Our #1 Competitor: Switching Friction (High Priority) Winning back the 26 trials lost to "No Decision" is our highest-leverage growth opportunity. * For Small Businesses (15 losses to "didn't get round to switching"): Build a 1-click import tool for existing client lists and past invoices (CSV/PDF scraper) and offer concierge migration on annual plans so switching takes 5 minutes, not a weekend. * For Freelancers (9 losses to spreadsheets/Word): Launch an instant "Template-to-Invoice" onboarding flow where users generate and send their first live invoice in under 60 seconds before even setting up full account settings. * For Accountant Objections (4 losses to Xero): Build or highlight a clean, 1-click "Export for Xero/QuickBooks" summary so SMBs can use Tallybook for daily invoicing without frustrating their accountants at tax time.

#### B. Audit Our Pricing Tiers to Stop the "Price" Losses (Quick Win) We shouldn't lose on price to a more expensive competitor. Let’s audit the 6 Freelancer/SMB deals lost to Billfront on price to see which specific feature pushed them into our $39 tier vs. Billfront’s $29 tier, and adjust our mid-tier packaging accordingly.

#### C. Make a Deliberate Call on Lightweight FX (Targeted) We do not need Billfront’s client portals or agency workflow tools. However, if we want to protect our $46 ARPU Agency segment and stop the bleeding on those 10 lost deals, we can integrate a lightweight third-party FX API (e.g., Open Exchange Rates) to populate live currency rates on invoices. * The rule: Only build lightweight live FX rates if it takes less than 2–3 weeks of engineering time. Do not follow Billfront into cross-border payment routing or agency project portals.

Let me know if you'd like to review the tier-packaging data together this week so we can lock the Q2 growth roadmap.

Claims the judge couldn’t find in the brief

  • Billfront accounted for 16 losses (35% of losses, 12% of total trials).
  • 10 of those Billfront losses were Multi-Currency Agencies needing live FX rates.
  • 17 lost deals (15 Small Business, 2 Agency) were due to 'didn't get round to switching'.
  • Total No Decision / Inertia losses are 26 (57% of all losses).
  • There were 111 combined trials (Freelancer + Small Business) out of 130.
  • We lose 63% more deals to friction during onboarding and switching than we do to Billfront.

Grades and run details

Decision model 68 · LLM judge 42
Decision model checks
  • failUses the supplied evidence correctly18%
  • passAddresses the actual decision45%
  • passRespects explicit constraints46%
  • partialIdentifies material uncertainty25%
  • partialAvoids unsupported claims49%
  • passProduces the required deliverable73%
  • passFinds the real competitor100%
  • passPins down where Billfront wins98%
  • passCompetes with what customers really choose100%
  • failGets the base of every number right18%
  • partialSeparates what a rival can copy48%
Run
Run
#1
API response time
24 s
Submitted
13 Oct 2026
Opus 5.5Claude

Usable with a quick edit11 of 11 checks passed

The analysis is data-backed, complete, and directly actionable. A PM could use it with minor wording edits and no need to re-verify evidence or reasoning.

Output

To: Dan Whitlock From: Growth PM Re: Billfront: where they actually hurt us, and what to do about it

---

The short version

Billfront isn't killing us. It is beating us in one segment, for one reason. Matching its whole feature set by Q2 would spend most of our roadmap on customers who aren't leaving us for Billfront. My recommendation is to build one targeted feature (multi-currency invoicing with live rates), deliberately skip the rest, and put equal effort into our real biggest loss: prospects who never switch to anything.

Who we actually lose to

Over the last two quarters, 130 trials became deals. We won 84 (65%) and lost 46.

Lost toDealsShare of lossesMain reason
No decision2554%Didn't get round to switching (16), stayed on spreadsheets/Word (9)
Billfront1737%Multi-currency with live rates (11), price (6)
Xero invoicing49%Their accountant uses something else

Our biggest competitor is inertia, not Billfront. Billfront accounts for 13% of all trials (17 of 130).

Where Billfront is a real threat, and where it isn't

It's a real threat in agencies. We lose 13 of 20 agency deals (35% win rate). Eleven of those losses went to Billfront, every one for the same reason: invoicing in several currencies with live rates. This is a single, clear feature gap, and it is the core of Billfront's paid offer. Their $29 Pro plan is built around it.

It isn't a threat in our core. Freelancers (77% win rate) and small businesses (63%) make up 92% of our revenue. Billfront took 6 deals from these segments, all on price. That is roughly 5% of core-segment deals.

The price losses are puzzling. Our entry plan is $9 and Billfront's is $12, so "price" probably means something else. Most likely a prospect needed a feature that's on our higher tier and is cheaper at Billfront, or compared our mid plan to their Starter. We should pull those six trial records before reacting to them.

How much revenue is at stake (each lost deal valued at its segment's average monthly revenue):

  • Billfront losses: about $610/month (multi-currency ~$505, price ~$105)
  • No-decision losses: about $525/month
  • Xero losses: about $95/month

By revenue, Billfront and inertia are roughly level. The difference is the fix. The Billfront gap is one feature. Inertia is spread across our largest segments and touches every trial.

Where Billfront is heading

Their public signals all point the same way, away from our core:

  • Positioning: the homepage says "Invoicing for global teams." Their case studies are three agencies and a consultancy, with no freelancers or local small businesses.
  • Launches: multi-currency in March, a client portal in June. Both are agency features.
  • Hiring: 4 engineers for "global payments" and 2 for "agency workflows."
  • Pricing: their plans go up to $59/month for teams. Our average agency customer pays us $46.

Billfront is moving upmarket into international agencies and cross-border payments. The risk to watch is global payments. If they make it easy to get paid across currencies, not just invoice in them, that could eventually appeal to our freelancers with overseas clients. For now there's no sign of that in our deal data.

What I recommend instead of matching everything

1. Build multi-currency invoicing with live rates for Q2 (match this one feature). This is the only Billfront feature that shows up in our losses, and it explains nearly all of them. Agencies are our highest-value customers per head ($46/month vs $24 and $11), even though the segment is small (240 customers, about 7% of revenue). Closing this gap protects them and blocks Billfront's main way in. Scope it tightly: multiple invoice currencies and live rates. Leave payments for later.

2. Don't build the client portal, team tier, or global payments yet. No lost deal cites any of them. Building them means chasing Billfront's strategy rather than our customers. We should re-check this every quarter against new loss data and their hiring.

3. Attack "no decision" with the same priority as #1. This is 25 of 46 losses. Two specific fixes: - Spreadsheet/Word import for freelancers (9 losses). Let them upload their existing template or client list and send a first invoice in minutes. - Switching help for small businesses (14 losses). Offer guided migration, "import your last 10 invoices," and trial nudges tied to their next billing date. Because these improve every trial, they also help us in the segments that make up most of our revenue.

4. Small bet on Xero (4 losses). Check whether a Xero sync, or simply marketing the one we have, gives accountants a reason to say yes.

5. Diagnose the six price losses before changing pricing.

Caveats

This is 130 deals over two quarters, with one recorded reason per loss. The revenue figures are rough, monthly, and per deal. The direction is clear enough to act on, but we should keep tagging loss reasons and re-run this analysis in a quarter.

If it helps, I can turn #1 and #3 into a Q2 roadmap proposal with effort estimates.

Grades and run details

Decision model 95 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly43%
  • passAddresses the actual decision56%
  • passRespects explicit constraints56%
  • passIdentifies material uncertainty82%
  • passAvoids unsupported claims55%
  • passProduces the required deliverable77%
  • passFinds the real competitor100%
  • passPins down where Billfront wins97%
  • passCompetes with what customers really choose100%
  • passGets the base of every number right38%
  • partialSeparates what a rival can copy48%
Run
Run
#1
Time to output
46 s
Submitted
5 Oct 2026
Gemini 3.5 Flash-LiteGemini

Needs more than a quick edit2 of 11 checks passed

The core data analysis is wrong, making the entire competitive assessment unreliable. A PM would need to re-analyze the deals file from scratch.

Output

MEMORANDUM

TO: Dan Whitlock, CEO FROM: Growth PM DATE: October 24, 2023 SUBJECT: Competitive Analysis: Tallybook vs. Billfront & Q2 Growth Strategy

---

Dan,

You noted that "Billfront is killing us" and asked for a feature match by Q2. I’ve dug into our last two quarters of deal data (trials T-900 through T-1029) and analyzed Billfront’s public footprint.

The data tells a surprising story: Billfront is not killing us broadly. In fact, we are winning the vast majority of our trials. However, Billfront is systematically poaching a very specific, high-value segment.

Below is our competitive reality check, a feature comparison, and a smarter Q2 plan than simply copying their roadmap.

---

1. Who We Really Lose To (The Data)

Out of 130 trial outcomes analyzed from the last two quarters, 74 trials were won (57%) and 56 were lost (43%).

When we lose, where do those deals actually go? * No Decision / Inertia (38 lost / 68% of losses): * Stayed on spreadsheets / Word templates: 16 losses. * Didn't get round to switching: 22 losses. * Takeaway: Our biggest enemy isn't a competitor; it's friction and customer apathy. * Xero (4 losses / 7% of losses): Small businesses whose accountants forced them onto an enterprise ledger. * Billfront (14 losses / 25% of losses): * Price: 4 freelancer losses (lost to their $12 entry tier vs our $9–$11 range). * Needs invoicing in several currencies with live rates: 10 agency losses.

The Verdict: Billfront is only a threat in the Multi-Currency Agency segment. They are positioning themselves as a vertical tool for global agencies, while we are trying to be a generalist tool.

---

2. Feature-by-Feature Comparison: Tallybook vs. Billfront

Feature / CapabilityTallybook ($9–$39/mo)Billfront ($12–$59/mo)Where Billfront is a Real ThreatWhere It’s Noise
:---:---:---:---:---
Core InvoicingRobust, clean templatesStandard invoicingNeutralBoth handle basic billing effortlessly.
Pricing & Packaging$9 (Freelancer), $24 (SMB), $46 (Agency ARPU)$12 (Starter), $29 (Pro), $59 (Team)Real Threat: Their $12 tier undercuts our SMB/Freelancer fence for price-sensitive buyers.
Multi-Currency & Live FXBasic manual entry / limited supportNative live FX rates & multi-currencyLethal Threat: 100% of our agency losses cite this exact feature.
Client PortalBasic payment link viewAdvanced portal (invoices, history, downloads)MinorNice-to-have, but didn't cause any documented churn/loss.
Global Payments / TeamsSingle-user focus, standard gatewaysLocalization, team workflows, agency toolsFuture Threat: Their current hiring (4 engineers for global payments, 2 for agency workflows) proves they are doubling down here.

---

3. Where Billfront is Heading

Based on their job postings and recent launches (client portal in June, multi-currency in March), Billfront is moving upmarket into localized, multi-entity agency operations. They are deliberately abandoning the low-end freelancer market to capture high-value agencies paying $59+/month.

If we spend Q2 blindly matching their features (like building a complex client portal or team permissions), we will burn engineering cycles copying a product that is deliberately scaling away from our bread-and-butter.

---

4. Strategic Recommendation: What We Should Do Instead of Matching

Our revenue is anchored by 5,800 Freelancers ($11 ARPU) and 3,100 Small Businesses ($24 ARPU). Our 240 Agencies ($46 ARPU) are nice, but they represent a fraction of our base.

Instead of playing catch-up to Billfront in a market we aren't optimized to win, we should execute a three-part growth plan for Q2:

#### A. Surrender the Multi-Currency Agency Segment (Deliberately) Stop trying to win agencies that demand live FX rates. Building a multi-currency engine requires heavy compliance, banking integrations, and engineering overhead to chase 10 lost deals a quarter. Let Billfront have them.

#### B. Fix the Real Leaks: Inertia and Onboarding (High ROI) Since 68% of our losses are to "No Decision," our biggest growth lever is reducing time-to-first-invoice for people stuck on spreadsheets. * Action: Build a 1-click CSV importer for past invoices and client lists in Q2. If we can convert half of the "spreadsheet/word template" drop-offs, we instantly capture 8+ new accounts a quarter without touching our pricing.

#### C. Protect the SMB Flank with Smart Packaging Billfront is picking off price-sensitive freelancers with their $12 tier. * Action: Introduce a streamlined "Solo" tier at $10/mo with zero friction, and double down on Tallybook's core differentiator: speed and simplicity for domestic small businesses and freelancers who don't care about global FX rates, they just want to get paid fast.

Summary: Billfront isn't killing us—they are just taking the multi-currency headaches off our hands. Let’s focus Q2 engineering power on wiping out spreadsheet inertia rather than chasing Billfront's agency ghost.

Best, Growth PM, Tallybook

Claims the judge couldn’t find in the brief

  • Out of 130 trial outcomes, 74 trials were won (57%) and 56 were lost (43%).
  • No Decision / Inertia accounts for 38 lost deals (68% of losses): 16 stayed on spreadsheets, 22 didn't get round to switching.
  • Billfront took 14 losses (25% of losses): 4 on price, 10 on multi-currency.
  • Tallybook's core invoicing has robust, clean templates.
  • Tallybook's multi-currency support is basic manual entry / limited support.
  • Tallybook's client portal is a basic payment link view.
  • Tallybook has a single-user focus and standard payment gateways.
  • Building a multi-currency engine requires heavy compliance, banking integrations, and engineering overhead.
  • Billfront is deliberately abandoning the low-end freelancer market.
  • Billfront's $12 tier undercuts our SMB/Freelancer fence for price-sensitive buyers.

Grades and run details

Decision model 50 · LLM judge 17
Decision model checks
  • failUses the supplied evidence correctly28%
  • failAddresses the actual decision76%
  • passRespects explicit constraints19%
  • failIdentifies material uncertainty67%
  • failAvoids unsupported claims33%
  • passProduces the required deliverable52%
  • passFinds the real competitor97%
  • passPins down where Billfront wins48%
  • passCompetes with what customers really choose100%
  • failGets the base of every number right46%
  • partialSeparates what a rival can copy67%
Run
Run
#1
Time to output
7 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 9

Uses the supplied evidence correctlyWrongRightWrong
Gemini 4 Argon · API

Several key numbers are taken incorrectly from the source data (Billfront losses 16 instead of 17, inertia losses 26 instead of 25, agency Billfront losses 10 instead of 11, trial counts off).

Opus 5.5 · Claude

All factual claims about the current situation are directly supported by the supplied context or arithmetic from it.

Gemini 3.5 Flash-Lite · Gemini

Key numbers (wins, losses, breakdown by reason) are incorrect, and many feature claims about Tallybook are invented.

Addresses the actual decisionRightRightWrong
Gemini 4 Argon · API

Commits early to not matching Billfront feature-for-feature and instead recommends a three-part plan focusing on switching friction, with a conditional on lightweight FX.

Opus 5.5 · Claude

Commits early to a clear recommendation (build multi-currency, skip the rest, attack no decision) and states that re-checking quarterly against new data would change it.

Gemini 3.5 Flash-Lite · Gemini

Does not state what result or condition would change the recommendation.

Identifies material uncertaintyWrongRightWrong
Gemini 4 Argon · API

Does not name unknowns that could change the overall decision (e.g., if agency churn becomes critical), and only mentions narrow pricing and build-time uncertainties.

Opus 5.5 · Claude

Names specific unknowns (true reason for price losses, future global payments risk) and says to pull trial records and re-run analysis quarterly.

Gemini 3.5 Flash-Lite · Gemini

No specific unknowns that could change the decision are named, nor how they would be resolved.

Avoids unsupported claimsWrongRightWrong
Gemini 4 Argon · API

Presents inaccurate counts as facts (e.g., 16 Billfront losses, 17 'didn't switch' losses) and states an interpretation ('Billfront is pivoting upmarket') without clear hedging.

Opus 5.5 · Claude

Hypotheses about price losses are clearly labelled as 'probably' and 'most likely'; forecasts are presented as risks to watch, not facts.

Gemini 3.5 Flash-Lite · Gemini

Presents interpretations (e.g., 'deliberately abandoning the low-end market') and feature details as fact without support.

Produces the required deliverableRightRightMixed
Gemini 4 Argon · API

Is a memo to the CEO, within the word limit, and contains all requested sections; with corrected numbers it would be usable.

Opus 5.5 · Claude

The memo is complete, in the right form for the CEO, within the word limit, and actionable with light edits.

Gemini 3.5 Flash-Lite · Gemini

The analysis contains wrong numbers, so a PM cannot act on it without redoing the data work.

Finds the real competitorMixedRightMixed
Gemini 4 Argon · API

Calls inertia the biggest competitor but uses wrong counts (26 instead of 25) that undermine the accuracy of the conclusion.

Opus 5.5 · Claude

Counts 25 no-decision losses vs 17 Billfront, names inertia as the biggest competitor, and provides the numbers.

Gemini 3.5 Flash-Lite · Gemini

Incorrectly counts no-decision losses as 38 instead of 25, and misstates the percentage.

Pins down where Billfront winsMixedRightMixed
Gemini 4 Argon · API

Identifies concentration in multi-currency agencies but states 10 agency losses to Billfront when the data shows 11, so the count is incorrect.

Opus 5.5 · Claude

Shows Billfront's wins are concentrated in multi-currency agencies (11 of 17) and sizes the segment at 240 customers, $46/month, ~7% of revenue.

Gemini 3.5 Flash-Lite · Gemini

Miscounts Billfront losses and misstates the concentration; does not correctly size the segment's revenue against others with accurate numbers.

Gets the base of every number rightWrongRightWrong
Gemini 4 Argon · API

Multiple derived figures are off (Billfront 16/17, inertia 26/25, agency Billfront 10/11, trial totals 111/110) showing arithmetic errors.

Opus 5.5 · Claude

All percentages, win rates, and revenue figures are computed from the correct denominators and match the supplied data.

Gemini 3.5 Flash-Lite · Gemini

All derived percentages and counts are based on wrong win/loss totals and wrong sub-counts.

Separates what a rival can copyRightRightWrong
Gemini 4 Argon · API

Focuses the response on reducing switching friction, a durable advantage that is hard for a well-funded competitor to copy, while noting Billfront’s feature path.

Opus 5.5 · Claude

Separates the copyable feature gap (multi-currency) from the durable advantage of reducing switching friction for core customers, and anchors the response in the latter.

Gemini 3.5 Flash-Lite · Gemini

Does not separate copyable advantages from durable ones; mentions 'speed and simplicity' but doesn't analyze its defensibility.

All got right 2

Respects explicit constraintsRightRightRight
Gemini 4 Argon · API

Stays under 900 words, is a competitive analysis directed to the CEO, and does not propose a feature-by-feature matching plan.

Opus 5.5 · Claude

Delivers a competitive analysis memo to Dan within 900 words, addressing all requested topics without proposing a full feature-by-feature match.

Gemini 3.5 Flash-Lite · Gemini

The output is a memo under 900 words, addresses the CEO, and does not propose a feature-by-feature matching plan.

Competes with what customers really chooseRightRightRight
Gemini 4 Argon · API

Treats spreadsheets and 'did not switch' as a competitor, and draws a direction-of-travel insight from Billfront's hiring and positioning.

Opus 5.5 · Claude

Treats spreadsheets/manual work/no decision as a competitor and draws a direction-of-travel insight from Billfront's hiring and launches to recommend against chasing their roadmap.

Gemini 3.5 Flash-Lite · Gemini

Treats spreadsheets and inertia as the main competitor and draws a direction-of-travel insight from Billfront's hiring.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6.1 SolAPI93.295.82None
2Opus 5.5Claude88.6100.02None
3GPT-6 AstraChatGPT86.495.82None
4GPT-6 LunaAPI93.283.32None
5Sonnet 5.5API86.483.32None
6Haiku 5.5API81.879.22None
7Gemini 3.8 FlashAPI68.254.22None
8Gemini 4 ArgonAPI65.954.22None
9Gemini 3.5 Flash-LiteGemini54.525.02None

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6.1 Sol · API: task score 95, 14× the cheapestOpus 5.5 · Claude: task score 95, 75× the cheapestGPT-6 Astra · ChatGPT: task score 92, 87× the cheapestGPT-6 Luna · API: task score 86, the cheapestSonnet 5.5 · API: task score 83, 44× the cheapestHaiku 5.5 · API: task score 79, 3.0× the cheapestGemini 3.8 Flash · API: task score 60, 12× the cheapestGemini 4 Argon · API: task score 57, 23× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 38, 2.9× the cheapestGPT-6.1 Sol · APIOpus 5.5 · ClaudeGPT-6 Astra · ChatGPTGPT-6 Luna · APISonnet 5.5 · APIHaiku 5.5 · APIGemini 3.8 Flash · APIGemini 4 Argon · APIGemini 3.5 Flash-Lite · Gemini
Nobody beats these on both score and cost

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Deciding how to respond to competitors.

Why it matters

Feature grids hide the real competition, which is often the customer's current workaround, and the response that feels urgent is usually to copy everything.

What good looks like

  • Includes the status quo and non-obvious alternatives
  • Finds where losses really go, by segment
  • Separates advantages a rival can copy from ones it can't
  • Reads where competitors are heading

Deliberately not measured

  • Market sizing
Capability tested

Competitive insight

The failure we’re looking for

A feature checklist and a plan to match everything

Grading

Decision model and LLM judge, calibrated against a blind PM review