Tasks / Strategy

Competitor analysis

Can the model find where a competitor can actually hurt you, from win/loss and churn data, rather than list features?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 63% were usable with at most a quick edit.

Reliably right

Respects explicit constraints100% pass
Delivers a competitive analysis memo to Dan within 900 words, addressing all requested topics without proposing a full feature-by-feature match.
Opus 5.5 · Claude · The competitor isn't who you think
Reads where Crestline is heading100% pass
Uses Crestline's hiring (payroll, multi-entity, accountant partnerships) to infer direction, flags the partner-programme risk, and adjusts the response accordingly.
GPT-6 Astra · ChatGPT · Free forever, funded
Produces the required deliverable97% pass
The memo is complete, in the right form for the CEO, within the word limit, and actionable with light edits.
Opus 5.5 · Claude · The competitor isn't who you think

Where it slips

Uses the supplied evidence correctly56% pass
Two factual claims are wrong: the number of unconnected multi-user accounts going to Crestline is 40, not 42, and the number of connected 6-20 cancellations is 5, not 4.
Sonnet 5.5 · API · Free forever, funded
Gets the base of every number right61% pass
Derived figures for unconnected multi-user cancellations (42 vs actual 40) and connected 6-20 cancellations (4 vs actual 5) are incorrect.
Sonnet 5.5 · API · Free forever, funded
Separates what a rival can copy66% pass
Does not explicitly separate advantages a well-funded competitor could copy from those it couldn't, nor anchor the response in durable advantages with evidence.
GPT-6 Luna · API · The competitor isn't who you think

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

Competes with what customers really choose
Does the analysis treat the status quo (spreadsheets, manual work, doing nothing) as a competitor, and draw at least one implication from where competitors are investing rather than only from today's features?
Passes when Analyses the status quo, strengths included, and gives a direction-of-travel insight that affects the recommendation.
April DunfordLenny’s Podcast
Separates what a rival can copy
Does the analysis separate advantages a well-funded competitor could copy (features, price, being first) from ones it couldn't easily (switching costs, networks, relationships), and build the response on the latter?
Passes when Names which advantages are copyable and which aren't, with evidence, and anchors the response in the durable ones.
Hamilton HelmerLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for growth at Tallybook. Our CEO, Dan Whitlock, wants a feature-by-feature comparison against Billfront and a plan to match everything they have by Q2 (his note is below). Write him a competitive analysis of no more than 900 words: who we really lose to, where Billfront is a real threat and where it isn't, where it's heading, and what we should do instead of (or as well as) matching it. The files are attached.

What the model was given5 items: About Tallybook, ceo_note.md, deals_last_two_quarters.csv (trials that became deals, won or lost), customers_by_segment.md, billfront_public_info.md

About TallybookInvoicing and payments for freelancers and small businesses. Plans from $9 to $39 a month.

ceo_note.md1 lines · Download

From Dan (CEO): "Billfront is killing us. I want a full feature-by-feature comparison against them and a plan to match everything they have by Q2."

deals_last_two_quarters.csv (trials that became deals, won or lost)deal,segment,outcome,primary_reason,went_to T-900,Small business,won,, T-901,Freelancer,won,, T-902,Freelancer,won,, T-903,Small business,won,, T-904,Small business,lost,Accountant uses something else,Xero invoicing T-905,Small business,won,, T-906,Small business,lost,Didn't get round to switching,No decision T-907,Freelancer,won,, T-908,Freelancer,won,, T-909,Small business,won,, T-910,Freelancer,lost,Price,Billfront T-911,Freelancer,won,, T-912,Small business,lost,Price,Billfront T-913,Freelancer,won,, T-914,Freelancer,won,, T-915,Small business,won,, T-916,Freelancer,won,, T-917,Small business,won,, T-918,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-919,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-920,Freelancer,won,, T-921,Agency (multi-currency),won,, T-922,Small business,won,, T-923,Small business,won,, T-924,Small business,won,, T-925,Freelancer,won,, T-926,Freelancer,won,, T-927,Freelancer,won,, T-928,Small business,won,, T-929,Small business,lost,Didn't get round to switching,No decision T-930,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-931,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-932,Small business,won,, T-933,Small business,won,, T-934,Small business,won,, T-935,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-936,Agency (multi-currency),lost,Didn't get round to switching,No decision T-937,Small business,lost,Didn't get round to switching,No decision T-938,Agency (multi-currency),won,, T-939,Small business,lost,Didn't get round to switching,No decision T-940,Small business,won,, T-941,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-942,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-943,Freelancer,won,, T-944,Small business,won,, T-945,Small business,lost,Accountant uses something else,Xero invoicing T-946,Small business,won,, T-947,Small business,won,, T-948,Small business,won,, T-949,Small business,lost,Didn't get round to switching,No decision T-950,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-951,Freelancer,won,, T-952,Freelancer,won,, T-953,Freelancer,won,, T-954,Agency (multi-currency),won,, T-955,Small business,lost,Didn't get round to switching,No decision T-956,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-957,Freelancer,won,, T-958,Small business,won,, T-959,Freelancer,won,, T-960,Small business,lost,Didn't get round to switching,No decision T-961,Freelancer,won,, T-962,Small business,won,, T-963,Agency (multi-currency),won,, T-964,Small business,won,, T-965,Small business,won,, T-966,Small business,won,, T-967,Small business,lost,Didn't get round to switching,No decision T-968,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-969,Small business,won,, T-970,Freelancer,lost,Price,Billfront T-971,Small business,lost,Didn't get round to switching,No decision T-972,Small business,lost,Accountant uses something else,Xero invoicing T-973,Small business,lost,Accountant uses something else,Xero invoicing T-974,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-975,Small business,lost,Price,Billfront T-976,Freelancer,won,, T-977,Small business,lost,Didn't get round to switching,No decision T-978,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-979,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-980,Freelancer,won,, T-981,Small business,won,, T-982,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-983,Small business,won,, T-984,Small business,lost,Didn't get round to switching,No decision T-985,Freelancer,won,, T-986,Freelancer,won,, T-987,Small business,lost,Didn't get round to switching,No decision T-988,Small business,won,, T-989,Freelancer,won,, T-990,Freelancer,won,, T-991,Freelancer,won,, T-992,Agency (multi-currency),won,, T-993,Freelancer,won,, T-994,Freelancer,won,, T-995,Small business,won,, T-996,Small business,won,, T-997,Small business,won,, T-998,Small business,lost,Didn't get round to switching,No decision T-999,Freelancer,lost,Stayed on spreadsheets / Word templates,No decision T-1000,Agency (multi-currency),won,, T-1001,Small business,lost,Price,Billfront T-1002,Small business,won,, T-1003,Freelancer,won,, T-1004,Agency (multi-currency),lost,Didn't get round to switching,No decision T-1005,Freelancer,won,, T-1006,Small business,won,, T-1007,Freelancer,won,, T-1008,Small business,won,, T-1009,Freelancer,lost,Price,Billfront T-1010,Small business,won,, T-1011,Freelancer,won,, T-1012,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-1013,Freelancer,won,, T-1014,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-1015,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-1016,Freelancer,won,, T-1017,Freelancer,won,, T-1018,Freelancer,won,, T-1019,Small business,lost,Didn't get round to switching,No decision T-1020,Freelancer,won,, T-1021,Small business,won,, T-1022,Agency (multi-currency),won,, T-1023,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-1024,Agency (multi-currency),lost,Needs invoicing in several currencies with live rates,Billfront T-1025,Freelancer,won,, T-1026,Freelancer,won,, T-1027,Freelancer,won,, T-1028,Small business,won,, T-1029,Freelancer,won,,

customers_by_segment.md7 lines · Download

# Tallybook customers and revenue (September)

| Segment | Paying customers | Average revenue per month |
|---|---|---|
| Freelancer | 5,800 | $11 |
| Small business | 3,100 | $24 |
| Agency (multi-currency) | 240 | $46 |

billfront_public_info.md6 lines · Download

# Billfront (public pages, September)

- Pricing: $12/month Starter, $29/month Pro (multi-currency, live FX rates), $59/month Team.
- Homepage: "Invoicing for global teams". Case studies: three agencies, one consultancy.
- Recent launches: multi-currency invoicing (March), client portal (June).
- Job posts: 4 engineers for "global payments", 2 for "agency workflows".

What a strong answer doesThe answer key the graders mark against

Shows from the deals file that the biggest competitor is no decision: of 46 lost deals, 25 ended with the prospect staying on spreadsheets or never switching, 17 went to Billfront and 4 to Xero invoicing. Billfront's wins are concentrated in multi-currency agencies (11 of its 17), on a specific gap (invoicing in several currencies with live rates); elsewhere it won only 6 deals, mostly on price. Agencies are a small segment (240 customers at about $46 a month, about $11,000 MRR, against about $64,000 from freelancers and $74,000 from small businesses), and Billfront is heading further into them (global payments and agency workflow hiring). Recommends against matching everything: fix switching friction for small businesses who 'didn't get round to it' (imports, setup help, accountant tie-ins), make a deliberate call on multi-currency (build it if agencies matter strategically, or cede the segment), and keep the advantages Billfront can't easily copy. Gives a short comparison only where it matters.

Critical failures (cap the score at 40)

Recommends matching Billfront feature for feature

Task

Mid-level · v1.0

The outputs

8 setups answered this brief.

Opus 5.5Claude98Usable with a quick edit GPT-6 LunaAPI94Usable with a quick edit GPT-6 AstraChatGPT94Usable with a quick edit Sonnet 5.5API94Usable with a quick edit GPT-6.1 SolAPI94Usable with a quick edit Haiku 5.5API92Usable with a quick edit Gemini 3.8 FlashAPI71Usable with a quick edit Gemini 3.5 Flash-LiteGemini35Needs more work

GPT-6 AstraChatGPT

Usable with a quick edit10 of 11 checks passed

The analysis is evidence-based and actionable; a PM could add a sentence on durable advantages without redoing any reasoning or re-checking evidence.

Output

To: Dan Whitlock Subject: Q2 competitive plan—beat the reasons we lose, not Billfront’s feature list

Dan—

Billfront is a real threat in multi-currency agencies, but the deal data does not support “Billfront is killing us” across our business. I recommend a Q2 plan focused on switching friction in our core market, with a gated investment in multi-currency—not blanket feature parity.

1. Who we actually lose to

Across 130 trials with recorded outcomes over the last two quarters, we won 84 and lost 46.

Destination	Lost deals	Share of losses	Recorded reason
---	---:	---:	---
No decision	25	54%	16 never switched; 9 stayed with spreadsheets/Word
Billfront	17	37%	11 multi-currency/live FX; 6 price
Xero invoicing	4	9%	Accountant uses another system

Our largest competitor is inertia. More deals stall or retain manual workflows than go to Billfront.

The segment split matters:

Freelancers and small businesses: 110 opportunities, 77 wins, and only six losses to Billfront—all attributed to price.
Multi-currency agencies: 20 opportunities, seven wins, 11 losses to Billfront, and two no-decisions. Billfront captures 55% of these opportunities and 85% of their losses.

These are counts from recorded trial outcomes—not market share, churn, or revenue-weighted losses. We lack deal values and quarter-by-quarter breakdowns, so cannot conclude that competitive losses are accelerating.

2. Where Billfront threatens us—and where it doesn’t

Our installed base remains overwhelmingly freelancers and small businesses: 97% of paying customers and 93% of monthly revenue, using September customer counts and ARPU. Agencies contribute approximately $11,040 of $149,240 monthly revenue.

Billfront is therefore a concentrated segment threat, not yet a demonstrated broad-based threat to our core. However, agencies represent 15% of this opportunity sample versus 3% of current customers. Ignoring them could constrain an emerging growth segment.

The defensible comparison from the supplied evidence is:

Dimension	Billfront	Tallybook / implication
Subscription pricing	$12 Starter; $29 Pro; $59 Team	$9–$39 plans. Billfront is not universally cheaper; investigate the actual packages compared in six price losses.
Multi-currency and live FX	Included in Pro; multi-currency launched in March	Eleven losses establish an unmet customer requirement. This is the strongest case for targeted parity.
Client portal	Launched in June	Tallybook capability is unspecified. No recorded loss cites it; validate demand before prioritizing.
Team/agency capabilities	Team tier; agency-focused hiring	Detailed functionality is unknown. Neither the plan name nor hiring proves specific feature advantages.
Positioning	“Invoicing for global teams”; three agency and one consultancy case studies	Tallybook serves freelancers and small businesses. Their positioning overlaps most strongly with our agency segment.

A complete feature-by-feature audit requires product access and our own capability inventory. We should do a bounded audit, but not turn every discovered difference into a roadmap commitment.

3. Where Billfront is heading

The evidence points toward cross-border payments and agency workflows: multi-currency is already shipped, four engineering roles target global payments, and two target agency workflows. The portal could support deeper agency-client collaboration.

That suggests a specialist competitor becoming stronger in international service businesses—not necessarily a competitor optimizing for solo freelancers. Hiring indicates direction, not a confirmed launch schedule. Matching today’s list could leave us following their next list while neglecting our larger conversion problem.

4. What we should do by Q2

First: make switching easier for our core customers. Interview recent no-decision prospects, then test guided setup, spreadsheet/client imports, reusable invoice templates, and assisted migration where the workflow audit shows gaps. Sell the outcome—getting the first invoice sent and paid with less effort—not feature volume. Measure trial-to-paid conversion, time to first invoice, first payment, and no-decision losses against a baseline or control.

Second: gate a focused multi-currency investment. Validate requirements with lost agencies and our seven agency wins: supported currencies, live-rate handling, settlement needs, and willingness to pay. Estimate build, provider, compliance, and support costs. Commit to a scoped Q2 release only if expected incremental contribution and retention justify it versus core activation work. Do not bundle a portal or full Team suite into that decision.

Third: diagnose price before cutting it. Review the six price losses for required features, chosen Billfront tier, discounts, and perceived value. Test clearer packaging or targeted offers rather than an across-the-board reduction. Reconcile agency ARPU of $46 with our stated $39 plan ceiling before modeling returns.

Fourth: reduce accountant-driven exclusions. Investigate the four Xero losses and test whether accountant-friendly exports or interoperability can preserve Tallybook as the invoicing layer.

Decision requested: replace “match everything by Q2” with a measured conversion program and a gated multi-currency bet. Keep a monthly competitor review, but fund work according to customer demand and incremental economics—not checklist completeness.

Grades and run details

Decision model 95 · LLM judge 92

Decision model checks

passUses the supplied evidence correctly44%
passAddresses the actual decision85%
passRespects explicit constraints55%
passIdentifies material uncertainty99%
passAvoids unsupported claims64%
passProduces the required deliverable48%
passFinds the real competitor100%
passPins down where Billfront wins100%
passCompetes with what customers really choose100%
passGets the base of every number right36%
partialSeparates what a rival can copy56%

Run

Run: #1
Time to output: 44 s
Submitted: 5 Oct 2026

GPT-6.1 SolAPI

Usable with a quick edit10 of 11 checks passed

The analysis is evidence-based, actionable, and requires no re-checking of facts or reasoning; a PM could use it with light edits.

Output

Competitive analysis: Billfront is a targeted threat—not our biggest source of losses

To: Dan Whitlock From: Growth PM

Recommendation: Do not commit to matching every Billfront feature by Q2. Commit to reducing switching friction in our core market, testing price objections, and validating a focused multi-currency response. Maintain a competitive feature inventory, but let customer evidence—not competitor launches—drive delivery.

1. Who we actually lose to

Across the 130 trial opportunities in the last two quarters, we won 84 and lost 46:

Loss destination	Deals	Share of losses	Recorded reason
---	---:	---:	---
No decision	25	54%	16 never switched; 9 stayed with spreadsheets/Word
Billfront	17	37%	11 needed multi-currency/live rates; 6 cited price
Xero invoicing	4	9%	Accountant preference

Our largest competitor is inertia. More opportunities stayed with their existing process than chose Billfront. Feature parity does not directly address those losses.

Billfront’s wins are concentrated: 11 of its 17 wins came from multi-currency agencies. Among the 110 freelancer and small-business opportunities, only six went to Billfront—all citing price.

These are trial-opportunity outcomes, not evidence of customer churn or overall market share. We also lack deal values, so counts alone cannot establish the revenue impact.

2. Where Billfront is—and is not—a real threat

Real threat: international agencies. Of 20 agency opportunities, we won seven, lost 11 to Billfront over multi-currency/live rates, and lost two to no decision. That is a specific, repeated capability gap, not a vague brand problem.

Agencies are a small but valuable segment: 240 paying customers generate approximately $11,040 monthly revenue, or 7.4% of the total, at the highest segment average of $46 per month. Protecting this segment merits investigation; pursuing it aggressively requires a deliberate market choice.

Limited demonstrated threat: our core market. Freelancers and small businesses represent 97.4% of paying customers and 92.6% of monthly revenue. Their recorded Billfront losses concern price, not missing portals or team workflows.

That does not mean price is objectively lower: Tallybook starts at $9 versus Billfront’s $12. We need to understand required tiers, perceived value, discounts, and payment costs before changing prices.

Separate threat: accountant-led selection. Four small businesses chose Xero because their accountant used it. Matching Billfront does nothing to resolve that workflow preference.

3. Feature comparison: what the evidence supports

A full parity assessment is not possible from these files: we lack Tallybook’s feature inventory and detailed Billfront documentation.

Area	Billfront evidence	Tallybook evidence / implication
Invoicing	Public positioning and launches confirm offering	Core Tallybook offering; workflow-level parity unverified
Payments	“Global payments” hiring	Core Tallybook offering; global capabilities unverified
Pricing	$12 Starter; $29 Pro; $59 Team	$9–$39 plans; six price losses require tier-level investigation
Multi-currency + live FX	Included in $29 Pro; launched March	Eleven losses establish an unmet requirement; exact existing coverage needs audit
Client portal	Launched June	Availability unknown; no recorded loss cites it
Team/agency workflows	Team plan; agency-workflow hiring	Detailed features unknown; no evidenced parity requirement yet

We should complete this inventory through product testing and documentation—not turn unknowns into assumed deficiencies.

4. Where Billfront appears to be heading

“Invoicing for global teams,” three agency case studies, one consultancy case study, and hiring for global payments and agency workflows point toward cross-border agency operations, rather than primarily low-cost freelancer invoicing.

The multi-currency launch and portal fit that direction. Hiring suggests possible investment in global payment capabilities and deeper agency workflows, but does not confirm particular upcoming features.

Following that trajectory wholesale would move us toward a different customer and potentially greater operational complexity.

5. What we should deliver by Q2

First: establish the facts. Product and Growth should audit both products, interview the 17 Billfront losses and a sample of no-decision opportunities, and identify the exact currency, payment, migration, and price requirements. Validate engineering scope before promising parity.

Prioritize core conversion. Test guided setup, invoice/customer import, template migration, and assisted switching. Measure first-invoice activation, time to first invoice, and trial-to-paid conversion against a control—not just onboarding completion.

Address price without a blanket cut. Test clearer tier/value messaging and targeted offers for price-sensitive prospects. Judge success on paid conversion, retained revenue, and margin.

Gate a focused FX investment. Scope multi-currency invoicing with live rates—not the entire global-payments stack. Proceed only if recoverable agency revenue, customer demand, and delivery cost justify it. Use agency design partners and explicit pilot adoption and retention criteria.

Investigate accountant compatibility. Determine whether exports, accountant access, or integrations would resolve the Xero losses before building.

Q2 commitment: validated competitive coverage, measured conversion experiments, and an evidence-based FX build decision—not indiscriminate feature parity.

Grades and run details

Decision model 95 · LLM judge 92

Decision model checks

passUses the supplied evidence correctly46%
passAddresses the actual decision89%
passRespects explicit constraints36%
passIdentifies material uncertainty100%
passAvoids unsupported claims64%
passProduces the required deliverable51%
passFinds the real competitor100%
passPins down where Billfront wins100%
passCompetes with what customers really choose100%
passGets the base of every number right44%
partialSeparates what a rival can copy61%

Run

Run: #1
API response time: 48 s
Submitted: 5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

All got wrong 1

Separates what a rival can copyWrongWrong

GPT-6 Astra · ChatGPT

Does not explicitly separate which advantages are copyable by a well-funded competitor and which are durable, nor anchor the response in the durable ones.

GPT-6.1 Sol · API

Does not explicitly separate copyable advantages (features, price) from durable ones (switching costs, relationships) or anchor the response in the latter.

All got right 10

Uses the supplied evidence correctlyRightRight

GPT-6 Astra · ChatGPT

All facts and figures are correctly taken from the supplied data, with no invented information about the current situation.

GPT-6.1 Sol · API

All factual claims are directly from the supplied files or derived by correct arithmetic.

Addresses the actual decisionRightRight

GPT-6 Astra · ChatGPT

Commits clearly to not matching everything, instead recommending a switching-friction program and a gated multi-currency investment, with conditions that would change the call.

GPT-6.1 Sol · API

Commits early to not matching everything, gives a clear alternative, and states conditions (recoverable revenue, demand, cost) that would change the call.

Respects explicit constraintsRightRight

GPT-6 Astra · ChatGPT

The output is a competitive analysis under 900 words, addressed to the CEO, and proposes what to do instead of matching, respecting the brief's request.

GPT-6.1 Sol · API

Output is a competitive analysis under 900 words, does not provide a full feature-by-feature matching plan, and is addressed to Dan.

Identifies material uncertaintyRightRight

GPT-6 Astra · ChatGPT

Names missing data (deal values, quarter breakdowns) and conditions for the multi-currency investment, and says how to resolve them through interviews, tests, and validation.

GPT-6.1 Sol · API

Names missing data (deal values, feature inventories, exact requirements) and says how to resolve them (audits, interviews, validation).

Avoids unsupported claimsRightRight

GPT-6 Astra · ChatGPT

Interpretations and forecasts are labelled as suggestions or conclusions from data, not presented as established fact.

GPT-6.1 Sol · API

Forecasts and interpretations are labelled as such (e.g., 'appears to be heading', 'suggests possible investment').

Produces the required deliverableRightRight

GPT-6 Astra · ChatGPT

The memo is complete, within the word limit, and directly usable by the CEO with actionable recommendations.

GPT-6.1 Sol · API

Delivers a complete, actionable competitive analysis memo for the CEO within the word limit.

Finds the real competitorRightRight

GPT-6 Astra · ChatGPT

Counts 25 no-decision losses versus 17 to Billfront and names inertia as the largest competitor.

GPT-6.1 Sol · API

Counts losses by destination, shows no decision as the largest (25 deals, 54%), and names inertia as the biggest competitor.

Pins down where Billfront winsRightRight

GPT-6 Astra · ChatGPT

Shows Billfront's wins are concentrated in multi-currency agencies (11 of 17) and sizes the segment's revenue at $11k of $149k.

GPT-6.1 Sol · API

Shows Billfront wins concentrated in multi-currency agencies (11 of 17) and sizes the segment's revenue ($11,040/month, 7.4% of total).

Competes with what customers really chooseRightRight

GPT-6 Astra · ChatGPT

Treats the status quo (spreadsheets, no decision) as a competitor and draws an implication from Billfront's hiring direction.

GPT-6.1 Sol · API

Treats spreadsheets/doing nothing as a competitor and gives a direction-of-travel insight (Billfront moving toward cross-border agency ops).

Gets the base of every number rightRightRight

GPT-6 Astra · ChatGPT

All percentages are correctly computed with clear denominators stated or implied.

GPT-6.1 Sol · API

All percentages and counts check out against the deals file and customer data, with bases stated where needed.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	Opus 5.5withClaude	86.4	100.0	2	None
2	GPT-6 AstrawithChatGPT	88.6	95.8	2	None
3	GPT-6.1 SolwithAPI	88.6	95.8	2	None
4	GPT-6 LunawithAPI	93.2	83.3	2	None
5	Sonnet 5.5withAPI	86.4	83.3	2	None
6	Haiku 5.5withAPI	84.1	79.2	2	None
7	Gemini 3.8 FlashwithAPI	68.2	54.2	2	None
8	Gemini 3.5 Flash-LitewithGemini	61.4	25.0	2	None

What you get for the cost

GPT-6.1 Sol · API scores within a point of Opus 5.5 · Claude at a fifth of the cost.

Best valueNobody beats these on both score and cost

Task score and relative cost by setup
Setup	Task score	Relative cost (cheapest = 1)
GPT-6 Luna · API	86	1.0×
Gemini 3.5 Flash-Lite · Gemini	41	2.9×
Haiku 5.5 · API	80	3.0×
Gemini 3.8 Flash · API	60	12×
GPT-6.1 Sol · API	93	14×
Sonnet 5.5 · API	83	44×
Opus 5.5 · Claude	94	75×
GPT-6 Astra · ChatGPT	93	87×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Deciding how to respond to competitors.

Why it matters

Feature grids hide the real competition, which is often the customer's current workaround, and the response that feels urgent is usually to copy everything.

What good looks like

Includes the status quo and non-obvious alternatives
Finds where losses really go, by segment
Separates advantages a rival can copy from ones it can't
Reads where competitors are heading

Deliberately not measured

Market sizing

Capability tested

Competitive insight

The failure we’re looking for

A feature checklist and a plan to match everything

Grading

Decision model and LLM judge, calibrated against a blind PM review

What AI gets right here, and what you’ll still have to catch

Reliably right

Where it slips

How it’s graded

Competes with what customers really choose

Separates what a rival can copy

Plus our standard checks

The tasks

The outputs

Output

1. Who we actually lose to

2. Where Billfront threatens us—and where it doesn’t

3. Where Billfront is heading

4. What we should do by Q2

Grades and run details

Decision model checks

Run

Output

Competitive analysis: Billfront is a targeted threat—not our biggest source of losses

1. Who we actually lose to

2. Where Billfront is—and is not—a real threat

3. Feature comparison: what the evidence supports

4. Where Billfront appears to be heading

5. What we should deliver by Q2

Grades and run details

Decision model checks

Run

Check by check

All got wrong 1

All got right 10

Results

What you get for the cost

About the task

The PM job

Why it matters

What good looks like

Deliberately not measured