Humanity’s Last PRD
Latest drop: Gemini 4 Argon · 13 Oct
Tasks / Strategy

Competitor analysis

Can the model find where a competitor can actually hurt you, from win/loss and churn data, rather than list features?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 56% were usable with at most a quick edit.

Reliably right

  1. Respects explicit constraints100% pass
    Delivers a competitive analysis memo to Dan within 900 words, addressing all requested topics without proposing a full feature-by-feature match.
    Opus 5.5 · Claude · The competitor isn't who you think
  2. Reads where Crestline is heading100% pass
    It reads Crestline's job ads as signalling payroll, mid-market, and possible accountant partnerships, and adjusts the response accordingly.
    GPT-6.1 Sol · API · Free forever, funded
  3. Produces the required deliverable97% pass
    The memo is complete, in the right form for the CEO, within the word limit, and actionable with light edits.
    Opus 5.5 · Claude · The competitor isn't who you think

Where it slips

  1. Uses the supplied evidence correctly56% pass
    Two factual claims are wrong: the number of unconnected multi-user accounts going to Crestline is 40, not 42, and the number of connected 6-20 cancellations is 5, not 4.
    Sonnet 5.5 · API · Free forever, funded
  2. Gets the base of every number right63% pass
    Derived figures for unconnected multi-user cancellations (42 vs actual 40) and connected 6-20 cancellations (4 vs actual 5) are incorrect.
    Sonnet 5.5 · API · Free forever, funded
  3. Separates what a rival can copy68% pass
    Does not explicitly separate advantages a well-funded competitor could copy from those it couldn't, nor anchor the response in durable advantages with evidence.
    GPT-6 Luna · API · The competitor isn't who you think

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Competes with what customers really choose

    Does the analysis treat the status quo (spreadsheets, manual work, doing nothing) as a competitor, and draw at least one implication from where competitors are investing rather than only from today's features?

    Passes when Analyses the status quo, strengths included, and gives a direction-of-travel insight that affects the recommendation.

    April DunfordLenny’s Podcast

  2. Separates what a rival can copy

    Does the analysis separate advantages a well-funded competitor could copy (features, price, being first) from ones it couldn't easily (switching costs, networks, relationships), and build the response on the latter?

    Passes when Names which advantages are copyable and which aren't, with evidence, and anchors the response in the durable ones.

    Hamilton HelmerLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for growth at Tallybook. Our CEO, Dan Whitlock, wants a feature-by-feature comparison against Billfront and a plan to match everything they have by Q2 (his note is below). Write him a competitive analysis of no more than 900 words: who we really lose to, where Billfront is a real threat and where it isn't, where it's heading, and what we should do instead of (or as well as) matching it. The files are attached.

What the model was given5 items: About Tallybook, ceo_note.md, deals_last_two_quarters.csv (trials that became deals, won or lost), customers_by_segment.md, billfront_public_info.md
About TallybookInvoicing and payments for freelancers and small businesses. Plans from $9 to $39 a month.
ceo_note.md1 lines · Download
From Dan (CEO): "Billfront is killing us. I want a full feature-by-feature comparison against them and a plan to match everything they have by Q2."
deals_last_two_quarters.csv (trials that became deals, won or lost)deal,segment,outcome,primary_reason,went_to T-900,Small business,won,, T-901,Freelancer,won,, T-902,Freelancer,won,, T-903,Small business,won,, T-904,Small business,lost,Accountant uses something else,Xero invoicing T-905,Small business,won,, T-906,Small business,lost,Didn't get round to switching,No decision T-907,Freelancer,won,, T-908,Freelancer,won,, T-909,Small business,won,, T-910,Freelancer,lost,Price,Billfront T-911,Freelancer,won,, T-912,Small business,lost,Price,Billfront T-913,Freelancer,won,, T-914,Freelancer,won,, T-915,Small business,won,, T-916,Freelancer,won,, T-917,Small business,won,, T-918,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-919,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-920,Freelancer,won,, T-921,Agency (multi-currency),won,, T-922,Small business,won,, T-923,Small business,won,, T-924,Small business,won,, T-925,Freelancer,won,, T-926,Freelancer,won,, T-927,Freelancer,won,, T-928,Small business,won,, T-929,Small business,lost,Didn't get round to switching,No decision T-930,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-931,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-932,Small business,won,, T-933,Small business,won,, T-934,Small business,won,, T-935,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-936,Agency (multi-currency),lost,Didn't get round to switching,No decision T-937,Small business,lost,Didn't get round to switching,No decision T-938,Agency (multi-currency),won,, T-939,Small business,lost,Didn't get round to switching,No decision T-940,Small business,won,, T-941,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-942,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-943,Freelancer,won,, T-944,Small business,won,, T-945,Small business,lost,Accountant uses something else,Xero invoicing T-946,Small business,won,, T-947,Small business,won,, T-948,Small business,won,, T-949,Small business,lost,Didn't get round to switching,No decision T-950,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-951,Freelancer,won,, T-952,Freelancer,won,, T-953,Freelancer,won,, T-954,Agency (multi-currency),won,, T-955,Small business,lost,Didn't get round to switching,No decision T-956,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-957,Freelancer,won,, T-958,Small business,won,, T-959,Freelancer,won,, T-960,Small business,lost,Didn't get round to switching,No decision T-961,Freelancer,won,, T-962,Small business,won,, T-963,Agency (multi-currency),won,, T-964,Small business,won,, T-965,Small business,won,, T-966,Small business,won,, T-967,Small business,lost,Didn't get round to switching,No decision T-968,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-969,Small business,won,, T-970,Freelancer,lost,Price,Billfront T-971,Small business,lost,Didn't get round to switching,No decision T-972,Small business,lost,Accountant uses something else,Xero invoicing T-973,Small business,lost,Accountant uses something else,Xero invoicing T-974,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-975,Small business,lost,Price,Billfront T-976,Freelancer,won,, T-977,Small business,lost,Didn't get round to switching,No decision T-978,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-979,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-980,Freelancer,won,, T-981,Small business,won,, T-982,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-983,Small business,won,, T-984,Small business,lost,Didn't get round to switching,No decision T-985,Freelancer,won,, T-986,Freelancer,won,, T-987,Small business,lost,Didn't get round to switching,No decision T-988,Small business,won,, T-989,Freelancer,won,, T-990,Freelancer,won,, T-991,Freelancer,won,, T-992,Agency (multi-currency),won,, T-993,Freelancer,won,, T-994,Freelancer,won,, T-995,Small business,won,, T-996,Small business,won,, T-997,Small business,won,, T-998,Small business,lost,Didn't get round to switching,No decision T-999,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-1000,Agency (multi-currency),won,, T-1001,Small business,lost,Price,Billfront T-1002,Small business,won,, T-1003,Freelancer,won,, T-1004,Agency (multi-currency),lost,Didn't get round to switching,No decision T-1005,Freelancer,won,, T-1006,Small business,won,, T-1007,Freelancer,won,, T-1008,Small business,won,, T-1009,Freelancer,lost,Price,Billfront T-1010,Small business,won,, T-1011,Freelancer,won,, T-1012,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-1013,Freelancer,won,, T-1014,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-1015,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-1016,Freelancer,won,, T-1017,Freelancer,won,, T-1018,Freelancer,won,, T-1019,Small business,lost,Didn't get round to switching,No decision T-1020,Freelancer,won,, T-1021,Small business,won,, T-1022,Agency (multi-currency),won,, T-1023,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-1024,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-1025,Freelancer,won,, T-1026,Freelancer,won,, T-1027,Freelancer,won,, T-1028,Small business,won,, T-1029,Freelancer,won,,
customers_by_segment.md7 lines · Download
# Tallybook customers and revenue (September)

| Segment | Paying customers | Average revenue per month |
|---|---|---|
| Freelancer | 5,800 | $11 |
| Small business | 3,100 | $24 |
| Agency (multi-currency) | 240 | $46 |
billfront_public_info.md6 lines · Download
# Billfront (public pages, September)

- Pricing: $12/month Starter, $29/month Pro (multi-currency, live FX rates), $59/month Team.
- Homepage: "Invoicing for global teams". Case studies: three agencies, one consultancy.
- Recent launches: multi-currency invoicing (March), client portal (June).
- Job posts: 4 engineers for "global payments", 2 for "agency workflows".
What a strong answer doesThe answer key the graders mark against

Shows from the deals file that the biggest competitor is no decision: of 46 lost deals, 25 ended with the prospect staying on spreadsheets or never switching, 17 went to Billfront and 4 to Xero invoicing. Billfront's wins are concentrated in multi-currency agencies (11 of its 17), on a specific gap (invoicing in several currencies with live rates); elsewhere it won only 6 deals, mostly on price. Agencies are a small segment (240 customers at about $46 a month, about $11,000 MRR, against about $64,000 from freelancers and $74,000 from small businesses), and Billfront is heading further into them (global payments and agency workflow hiring). Recommends against matching everything: fix switching friction for small businesses who 'didn't get round to it' (imports, setup help, accountant tie-ins), make a deliberate call on multi-currency (build it if agencies matter strategically, or cede the segment), and keep the advantages Billfront can't easily copy. Gives a short comparison only where it matters.

Critical failures (cap the score at 40)
  • Recommends matching Billfront feature for feature
Task

Mid-level · v1.2

The outputs

9 setups answered this brief.

Opus 5.5Claude98Usable with a quick editGPT-6 LunaAPI94Usable with a quick editGPT-6 AstraChatGPT94Usable with a quick editSonnet 5.5API94Usable with a quick editGPT-6.1 SolAPI94Usable with a quick editHaiku 5.5API92Usable with a quick editGemini 3.8 FlashAPI73Usable with a quick editGemini 4 ArgonAPI52Needs more workGemini 3.5 Flash-LiteGemini31Needs more work
Gemini 4 ArgonAPI

Needs more than a quick edit5 of 11 checks passed

The output contains multiple numerical errors (Billfront loss count, no-decision count, trial count) that would require re-checking the data and redoing parts of the analysis. It cannot be used with just a quick edit.

Output

To: Dan Whitlock, CEO From: PM, Growth Subject: Billfront Competitive Analysis & Q2 Growth Plan

Dan,

Before building a Q2 roadmap to match Billfront feature-for-feature, I pulled our win/loss data from the last two quarters (130 trials) alongside our revenue breakdown and Billfront’s recent activity.

The data shows that Billfront is not killing us in our core business. Matching their roadmap by Q2 would pull engineering away from our biggest revenue drivers to chase a small, non-core segment. Here is what is actually happening in our pipeline and what we should do instead.

---

1. Who We Really Lose To: Inertia, Not Billfront

Out of 130 trials over the last two quarters, we won 84 (65%) and lost 46. When we lose, it usually isn’t to a software competitor—it’s to status quo inertia:

  • No Decision / Inertia (26 losses — 57% of all losses):
  • 17 deals (15 Small Business, 2 Agency) simply "didn't get round to switching."
  • 9 deals (all Freelancers) "stayed on spreadsheets or Word templates."
  • Billfront (16 losses — 35% of losses, 12% of total trials):
  • 10 deals were Multi-Currency Agencies needing live FX rates.
  • 6 deals (3 Freelancer, 3 Small Business) cited price.
  • Xero Invoicing (4 losses — 9% of losses):
  • All 4 were Small Businesses whose "accountant uses something else."

The takeaway: We lose 63% more deals to friction during onboarding and switching than we do to Billfront.

---

2. Where Billfront Is a Real Threat—And Where It Isn't

Where they are a threat: Multi-Currency Agencies In the Agency segment, Billfront is beating us. Out of 19 Agency trials, we won 7 and lost 12—10 of which went directly to Billfront because they launched multi-currency invoicing with live FX rates in March.

However, we need to put this segment in perspective against our current monthly recurring revenue ($149.2k total MRR): * Freelancers: 5,800 customers @ $11 ARPU = $63,800/mo (43% of MRR) * Small Businesses: 3,100 customers @ $24 ARPU = $74,400/mo (50% of MRR) * Agencies (multi-currency): 240 customers @ $46 ARPU = $11,040/mo (7% of MRR)

Where they are NOT a threat: Our Core (93% of our revenue) Across Freelancers and Small Businesses (111 combined trials), we lost only 6 deals to Billfront—all on "price," not missing features.

Notably, our plans ($9–$39/mo) are actually cheaper at the entry and top tiers than Billfront’s ($12 Starter, $29 Pro, $59 Team). Losing 6 core deals on "price" indicates a tier-packaging friction—specifically, small businesses or freelancers hitting a paywall forcing them onto our $39 plan when Billfront’s $29 Pro tier covers their needs—not a product gap.

---

3. Where Billfront Is Heading

Billfront is pivoting upmarket to become a workflow and cross-border payments platform for agencies and consultancies: * Positioning: Their homepage now leads with "Invoicing for global teams," backed exclusively by agency and consultancy case studies. * Recent Launches: Multi-currency invoicing (March) and a client portal (June). * Hiring: They are recruiting 4 engineers for "global payments" and 2 for "agency workflows." * Pricing: Their new $59/month Team tier reflects an upmarket push.

If we try to match everything they have by Q2 (client portals, agency workflows, global payments infrastructure), we will spend the next two quarters building complex software for 7% of our customer base while neglecting the 93% who just want fast, simple invoicing.

---

4. What We Should Do Instead by Q2

Rather than a feature-for-feature clone of Billfront, I recommend a focused three-part plan for Q2:

#### A. Attack Our #1 Competitor: Switching Friction (High Priority) Winning back the 26 trials lost to "No Decision" is our highest-leverage growth opportunity. * For Small Businesses (15 losses to "didn't get round to switching"): Build a 1-click import tool for existing client lists and past invoices (CSV/PDF scraper) and offer concierge migration on annual plans so switching takes 5 minutes, not a weekend. * For Freelancers (9 losses to spreadsheets/Word): Launch an instant "Template-to-Invoice" onboarding flow where users generate and send their first live invoice in under 60 seconds before even setting up full account settings. * For Accountant Objections (4 losses to Xero): Build or highlight a clean, 1-click "Export for Xero/QuickBooks" summary so SMBs can use Tallybook for daily invoicing without frustrating their accountants at tax time.

#### B. Audit Our Pricing Tiers to Stop the "Price" Losses (Quick Win) We shouldn't lose on price to a more expensive competitor. Let’s audit the 6 Freelancer/SMB deals lost to Billfront on price to see which specific feature pushed them into our $39 tier vs. Billfront’s $29 tier, and adjust our mid-tier packaging accordingly.

#### C. Make a Deliberate Call on Lightweight FX (Targeted) We do not need Billfront’s client portals or agency workflow tools. However, if we want to protect our $46 ARPU Agency segment and stop the bleeding on those 10 lost deals, we can integrate a lightweight third-party FX API (e.g., Open Exchange Rates) to populate live currency rates on invoices. * The rule: Only build lightweight live FX rates if it takes less than 2–3 weeks of engineering time. Do not follow Billfront into cross-border payment routing or agency project portals.

Let me know if you'd like to review the tier-packaging data together this week so we can lock the Q2 growth roadmap.

Claims the judge couldn’t find in the brief

  • Billfront accounted for 16 losses (35% of losses, 12% of total trials).
  • 10 of those Billfront losses were Multi-Currency Agencies needing live FX rates.
  • 17 lost deals (15 Small Business, 2 Agency) were due to 'didn't get round to switching'.
  • Total No Decision / Inertia losses are 26 (57% of all losses).
  • There were 111 combined trials (Freelancer + Small Business) out of 130.
  • We lose 63% more deals to friction during onboarding and switching than we do to Billfront.

Grades and run details

Decision model 68 · LLM judge 42
Decision model checks
  • failUses the supplied evidence correctly18%
  • passAddresses the actual decision45%
  • passRespects explicit constraints46%
  • partialIdentifies material uncertainty25%
  • partialAvoids unsupported claims49%
  • passProduces the required deliverable73%
  • passFinds the real competitor100%
  • passPins down where Billfront wins98%
  • passCompetes with what customers really choose100%
  • failGets the base of every number right18%
  • partialSeparates what a rival can copy48%
Run
Run
#1
API response time
24 s
Submitted
13 Oct 2026
GPT-6 AstraChatGPT

Usable with a quick edit10 of 11 checks passed

The analysis is evidence-based and actionable; a PM could add a sentence on durable advantages without redoing any reasoning or re-checking evidence.

Output

To: Dan Whitlock Subject: Q2 competitive plan—beat the reasons we lose, not Billfront’s feature list

Dan—

Billfront is a real threat in multi-currency agencies, but the deal data does not support “Billfront is killing us” across our business. I recommend a Q2 plan focused on switching friction in our core market, with a gated investment in multi-currency—not blanket feature parity.

1. Who we actually lose to

Across 130 trials with recorded outcomes over the last two quarters, we won 84 and lost 46.

DestinationLost dealsShare of lossesRecorded reason
------:---:---
No decision2554%16 never switched; 9 stayed with spreadsheets/Word
Billfront1737%11 multi-currency/live FX; 6 price
Xero invoicing49%Accountant uses another system

Our largest competitor is inertia. More deals stall or retain manual workflows than go to Billfront.

The segment split matters:

  • Freelancers and small businesses: 110 opportunities, 77 wins, and only six losses to Billfront—all attributed to price.
  • Multi-currency agencies: 20 opportunities, seven wins, 11 losses to Billfront, and two no-decisions. Billfront captures 55% of these opportunities and 85% of their losses.

These are counts from recorded trial outcomes—not market share, churn, or revenue-weighted losses. We lack deal values and quarter-by-quarter breakdowns, so cannot conclude that competitive losses are accelerating.

2. Where Billfront threatens us—and where it doesn’t

Our installed base remains overwhelmingly freelancers and small businesses: 97% of paying customers and 93% of monthly revenue, using September customer counts and ARPU. Agencies contribute approximately $11,040 of $149,240 monthly revenue.

Billfront is therefore a concentrated segment threat, not yet a demonstrated broad-based threat to our core. However, agencies represent 15% of this opportunity sample versus 3% of current customers. Ignoring them could constrain an emerging growth segment.

The defensible comparison from the supplied evidence is:

DimensionBillfrontTallybook / implication
Subscription pricing$12 Starter; $29 Pro; $59 Team$9–$39 plans. Billfront is not universally cheaper; investigate the actual packages compared in six price losses.
Multi-currency and live FXIncluded in Pro; multi-currency launched in MarchEleven losses establish an unmet customer requirement. This is the strongest case for targeted parity.
Client portalLaunched in JuneTallybook capability is unspecified. No recorded loss cites it; validate demand before prioritizing.
Team/agency capabilitiesTeam tier; agency-focused hiringDetailed functionality is unknown. Neither the plan name nor hiring proves specific feature advantages.
Positioning“Invoicing for global teams”; three agency and one consultancy case studiesTallybook serves freelancers and small businesses. Their positioning overlaps most strongly with our agency segment.

A complete feature-by-feature audit requires product access and our own capability inventory. We should do a bounded audit, but not turn every discovered difference into a roadmap commitment.

3. Where Billfront is heading

The evidence points toward cross-border payments and agency workflows: multi-currency is already shipped, four engineering roles target global payments, and two target agency workflows. The portal could support deeper agency-client collaboration.

That suggests a specialist competitor becoming stronger in international service businesses—not necessarily a competitor optimizing for solo freelancers. Hiring indicates direction, not a confirmed launch schedule. Matching today’s list could leave us following their next list while neglecting our larger conversion problem.

4. What we should do by Q2

First: make switching easier for our core customers. Interview recent no-decision prospects, then test guided setup, spreadsheet/client imports, reusable invoice templates, and assisted migration where the workflow audit shows gaps. Sell the outcome—getting the first invoice sent and paid with less effort—not feature volume. Measure trial-to-paid conversion, time to first invoice, first payment, and no-decision losses against a baseline or control.

Second: gate a focused multi-currency investment. Validate requirements with lost agencies and our seven agency wins: supported currencies, live-rate handling, settlement needs, and willingness to pay. Estimate build, provider, compliance, and support costs. Commit to a scoped Q2 release only if expected incremental contribution and retention justify it versus core activation work. Do not bundle a portal or full Team suite into that decision.

Third: diagnose price before cutting it. Review the six price losses for required features, chosen Billfront tier, discounts, and perceived value. Test clearer packaging or targeted offers rather than an across-the-board reduction. Reconcile agency ARPU of $46 with our stated $39 plan ceiling before modeling returns.

Fourth: reduce accountant-driven exclusions. Investigate the four Xero losses and test whether accountant-friendly exports or interoperability can preserve Tallybook as the invoicing layer.

Decision requested: replace “match everything by Q2” with a measured conversion program and a gated multi-currency bet. Keep a monthly competitor review, but fund work according to customer demand and incremental economics—not checklist completeness.

Grades and run details

Decision model 95 · LLM judge 92
Decision model checks
  • passUses the supplied evidence correctly59%
  • passAddresses the actual decision52%
  • passRespects explicit constraints52%
  • passIdentifies material uncertainty98%
  • passAvoids unsupported claims73%
  • passProduces the required deliverable46%
  • passFinds the real competitor100%
  • passPins down where Billfront wins100%
  • passCompetes with what customers really choose100%
  • passGets the base of every number right43%
  • partialSeparates what a rival can copy56%
Run
Run
#1
Time to output
44 s
Submitted
5 Oct 2026
GPT-6 LunaAPI

Usable with a quick edit10 of 11 checks passed

The memo is clear, evidence-based, and actionable; a PM could use it with minor wording edits without rechecking data or redoing reasoning.

Output

To: Dan Whitlock Subject: Billfront: where we’re losing—and what to do by Q2

Bottom line: Billfront is a real, focused threat in multi-currency agency invoicing, and a smaller price competitor elsewhere. But it is not the main explanation for our losses overall. A plan to match every feature by Q2 would spread effort across threats the deal data doesn’t support.

Who we really lose to: In 130 trial deals from the last two quarters, we won 84 and lost 46. Billfront was named in 17 losses (37%): 11 for multi-currency invoicing with live FX, and six for price. The other 29 losses were to no decision (16), spreadsheets or Word templates (9), and an accountant’s preference for Xero (4). In other words, stalled switching and familiar alternatives together account for more losses than Billfront.

These are recorded reasons from trial deals, not a complete view of the market. But they argue for fixing activation and switching friction alongside competitive product work—not treating every loss as a feature gap.

Where Billfront is a real threat: The signal is concentrated in agencies. Of 20 agency multi-currency deals, 11 were lost to Billfront specifically because prospects needed invoicing in several currencies with live rates. That’s a clear, repeated unmet need. The segment is small today—240 of our 9,140 customers—but its $46 average monthly revenue makes up about 7% of Tallybook MRR. It merits focused investment.

Where it isn’t: Billfront is not demonstrably beating us across freelancers and small businesses on a broad product advantage. Six price losses matter, but our public starting price is lower ($9 vs. Billfront’s $12); we don’t have enough plan-level detail to compare value or packaging. Nor should we assume we lack Billfront’s other features: the materials provided don’t document Tallybook’s feature set beyond invoicing and payments.

AreaWhat we can verify about BillfrontWhat we can verify about Tallybook
Pricing$12 Starter; $29 Pro; $59 TeamPlans from $9–$39; tier details unavailable
Multi-currency / live FXIncluded in Pro; cited in 11 lossesCapability not documented
Client portalLaunched in JuneCapability not documented
Global payments / agency workflowsHiring for both; these are signals of direction, not proof of shipped featuresCapability not documented

Where Billfront is heading: Its “invoicing for global teams” positioning, recent multi-currency and portal launches, and hiring across global payments and agency workflows point toward a broader agency platform. That makes the agency threat more likely to grow. Hiring is a roadmap signal, though—not confirmation those capabilities have shipped.

Recommendation: match the need, not the whole product.

  1. By Q2, validate and deliver a focused multi-currency offer for agencies. Start with discovery among lost and won agency prospects. If confirmed, prioritize multi-currency invoices and transparent, reliable live-rate handling; test packaging and willingness to pay before committing to a larger build. Track agency win rate and conversion, not feature count.
  2. Reduce the larger pool of avoidable losses. Improve trial onboarding, migration help, and prompts that get a first invoice sent. For Xero-driven losses, assess an integration or accountant-friendly workflow before building accounting features. These address observed losses without trying to replace established systems.
  3. Test price and packaging before cutting prices. The six Billfront price losses warrant follow-up on what customers compared and what value they perceived. Our lower entry price alone isn’t resolving those objections.
  4. Gate further parity work on evidence. Revisit portals, global payments, and agency workflows only if customer research, win/loss data, or retention shows they block adoption.

Decision: Approve a targeted agency multi-currency initiative plus switching and activation improvements—not a blanket commitment to match everything Billfront has by Q2.

Grades and run details

Decision model 95 · LLM judge 92
Decision model checks
  • passUses the supplied evidence correctly55%
  • passAddresses the actual decision66%
  • passRespects explicit constraints70%
  • passIdentifies material uncertainty91%
  • passAvoids unsupported claims82%
  • passProduces the required deliverable64%
  • passFinds the real competitor83%
  • passPins down where Billfront wins92%
  • passCompetes with what customers really choose100%
  • passGets the base of every number right52%
  • partialSeparates what a rival can copy77%
Run
Run
#1
API response time
24 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 7

Uses the supplied evidence correctlyWrongRightRight
Gemini 4 Argon · API

Several key numbers are taken incorrectly from the source data (Billfront losses 16 instead of 17, inertia losses 26 instead of 25, agency Billfront losses 10 instead of 11, trial counts off).

GPT-6 Astra · ChatGPT

All facts and figures are correctly taken from the supplied data, with no invented information about the current situation.

GPT-6 Luna · API

All factual claims about the current situation are directly supported by the supplied data or arithmetic.

Identifies material uncertaintyWrongRightRight
Gemini 4 Argon · API

Does not name unknowns that could change the overall decision (e.g., if agency churn becomes critical), and only mentions narrow pricing and build-time uncertainties.

GPT-6 Astra · ChatGPT

Names missing data (deal values, quarter breakdowns) and conditions for the multi-currency investment, and says how to resolve them through interviews, tests, and validation.

GPT-6 Luna · API

Names unknowns (lack of plan detail, Tallybook feature set, hiring as signal not shipped feature) and says how to resolve (discovery with agency prospects, gating further work on evidence).

Avoids unsupported claimsWrongRightRight
Gemini 4 Argon · API

Presents inaccurate counts as facts (e.g., 16 Billfront losses, 17 'didn't switch' losses) and states an interpretation ('Billfront is pivoting upmarket') without clear hedging.

GPT-6 Astra · ChatGPT

Interpretations and forecasts are labelled as suggestions or conclusions from data, not presented as established fact.

GPT-6 Luna · API

Interpretations are labelled as such (e.g., 'argue for', 'not demonstrably beating us'), and confident claims are backed by data.

Finds the real competitorMixedRightRight
Gemini 4 Argon · API

Calls inertia the biggest competitor but uses wrong counts (26 instead of 25) that undermine the accuracy of the conclusion.

GPT-6 Astra · ChatGPT

Counts 25 no-decision losses versus 17 to Billfront and names inertia as the largest competitor.

GPT-6 Luna · API

Counts losses by destination, shows 16 'didn't get round to switching' and 9 spreadsheets/Word templates, and names no decision as the biggest loss category.

Pins down where Billfront winsMixedRightRight
Gemini 4 Argon · API

Identifies concentration in multi-currency agencies but states 10 agency losses to Billfront when the data shows 11, so the count is incorrect.

GPT-6 Astra · ChatGPT

Shows Billfront's wins are concentrated in multi-currency agencies (11 of 17) and sizes the segment's revenue at $11k of $149k.

GPT-6 Luna · API

Shows Billfront wins concentrated in multi-currency agencies (11 of 17), and sizes the segment at 240 customers, ~7% of MRR.

Gets the base of every number rightWrongRightRight
Gemini 4 Argon · API

Multiple derived figures are off (Billfront 16/17, inertia 26/25, agency Billfront 10/11, trial totals 111/110) showing arithmetic errors.

GPT-6 Astra · ChatGPT

All percentages are correctly computed with clear denominators stated or implied.

GPT-6 Luna · API

All derived figures (130 deals, 84 won, 46 lost, 17 Billfront, 11 multi-currency, 6 price, 29 other, 7% MRR) check out against the data.

Separates what a rival can copyRightWrongWrong
Gemini 4 Argon · API

Focuses the response on reducing switching friction, a durable advantage that is hard for a well-funded competitor to copy, while noting Billfront’s feature path.

GPT-6 Astra · ChatGPT

Does not explicitly separate which advantages are copyable by a well-funded competitor and which are durable, nor anchor the response in the durable ones.

GPT-6 Luna · API

Does not explicitly separate advantages a well-funded competitor could copy from those it couldn't, nor anchor the response in durable advantages with evidence.

All got right 4

Addresses the actual decisionRightRightRight
Gemini 4 Argon · API

Commits early to not matching Billfront feature-for-feature and instead recommends a three-part plan focusing on switching friction, with a conditional on lightweight FX.

GPT-6 Astra · ChatGPT

Commits clearly to not matching everything, instead recommending a switching-friction program and a gated multi-currency investment, with conditions that would change the call.

GPT-6 Luna · API

Commits to a clear answer (targeted agency multi-currency plus switching improvements, not blanket matching) and states what would change it (further parity work gated on evidence).

Respects explicit constraintsRightRightRight
Gemini 4 Argon · API

Stays under 900 words, is a competitive analysis directed to the CEO, and does not propose a feature-by-feature matching plan.

GPT-6 Astra · ChatGPT

The output is a competitive analysis under 900 words, addressed to the CEO, and proposes what to do instead of matching, respecting the brief's request.

GPT-6 Luna · API

Output is a competitive analysis memo under 900 words, does not propose a full feature-by-feature matching plan, and addresses the CEO as requested.

Produces the required deliverableRightRightRight
Gemini 4 Argon · API

Is a memo to the CEO, within the word limit, and contains all requested sections; with corrected numbers it would be usable.

GPT-6 Astra · ChatGPT

The memo is complete, within the word limit, and directly usable by the CEO with actionable recommendations.

GPT-6 Luna · API

Delivers a complete competitive analysis memo to the CEO within the word limit, with actionable recommendations.

Competes with what customers really chooseRightRightRight
Gemini 4 Argon · API

Treats spreadsheets and 'did not switch' as a competitor, and draws a direction-of-travel insight from Billfront's hiring and positioning.

GPT-6 Astra · ChatGPT

Treats the status quo (spreadsheets, no decision) as a competitor and draws an implication from Billfront's hiring direction.

GPT-6 Luna · API

Treats spreadsheets/doing nothing as a competitor, and gives a direction-of-travel insight (Billfront hiring signals broader agency platform, making threat likely to grow).

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6.1 SolAPI93.295.82None
2Opus 5.5Claude88.6100.02None
3GPT-6 AstraChatGPT86.495.82None
4GPT-6 LunaAPI93.283.32None
5Sonnet 5.5API86.483.32None
6Haiku 5.5API81.879.22None
7Gemini 3.8 FlashAPI68.254.22None
8Gemini 4 ArgonAPI65.954.22None
9Gemini 3.5 Flash-LiteGemini54.525.02None

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6.1 Sol · API: task score 95, 14× the cheapestOpus 5.5 · Claude: task score 95, 75× the cheapestGPT-6 Astra · ChatGPT: task score 92, 87× the cheapestGPT-6 Luna · API: task score 86, the cheapestSonnet 5.5 · API: task score 83, 44× the cheapestHaiku 5.5 · API: task score 79, 3.0× the cheapestGemini 3.8 Flash · API: task score 60, 12× the cheapestGemini 4 Argon · API: task score 57, 23× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 38, 2.9× the cheapestGPT-6.1 Sol · APIOpus 5.5 · ClaudeGPT-6 Astra · ChatGPTGPT-6 Luna · APISonnet 5.5 · APIHaiku 5.5 · APIGemini 3.8 Flash · APIGemini 4 Argon · APIGemini 3.5 Flash-Lite · Gemini
Nobody beats these on both score and cost

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Deciding how to respond to competitors.

Why it matters

Feature grids hide the real competition, which is often the customer's current workaround, and the response that feels urgent is usually to copy everything.

What good looks like

  • Includes the status quo and non-obvious alternatives
  • Finds where losses really go, by segment
  • Separates advantages a rival can copy from ones it can't
  • Reads where competitors are heading

Deliberately not measured

  • Market sizing
Capability tested

Competitive insight

The failure we’re looking for

A feature checklist and a plan to match everything

Grading

Decision model and LLM judge, calibrated against a blind PM review