Tasks / Strategy

Price and package a product

Can the model price on the value each kind of customer gets, using real usage and willingness-to-pay data, and get existing customers there?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 56% were usable with at most a quick edit.

Reliably right

  1. Doesn't trust the price test100% pass
    Names the 59/41 split mismatch, holiday timing, and small sample; does not rest the decision on the test.
    Opus 5.5 · Claude · Unlimited pages, falling margins
  2. Plans the renewals100% pass
    Addresses each enterprise renewal with its date and note, including Copperfield in January, Northwind's tripled volume, and Arcadia's competitor evaluation.
    Opus 5.5 · Claude · Unlimited pages, falling margins
  3. Each tier for a different buyer94% pass
    Each tier names its buyer (startups, mid-market, enterprise) and is priced on distinct value: startups protected, mid-market overage on heavy users, enterprise per-page with unlimited seats.
    Opus 5.5 · Claude · Unlimited pages, falling margins

Where it slips

  1. Uses the supplied evidence correctly34% pass
    Total MRR is stated as $11,670 (actual $11,820), and median 'getting expensive' for small agencies is given as about $37 (actual $33) and for agencies as about $55 (actual $51).
    Haiku 5.5 · API · Fifteen dollars for everyone
  2. Gets the base of every number right41% pass
    MRR total and two segment‑level median getting‑expensive figures are incorrectly computed from the supplied data.
    Haiku 5.5 · API · Fifteen dollars for everyone
  3. Avoids unsupported claims52% pass
    Median getting‑expensive values and total MRR are presented as fact but are not supported by the data; these are facts about the current situation, not labelled as assumptions.
    Haiku 5.5 · API · Fifteen dollars for everyone

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Each tier for a different buyer

    Is each plan built for a distinct kind of buyer, with what that buyer values, rather than more-of-the-same at a higher price?

    Passes when Every tier names its buyer and the value it's priced on, and includes what that buyer needs and the lower tier's buyer doesn't.

  2. Priced on willingness to pay

    Are the proposed prices grounded in evidence of what each kind of customer would pay and the value they get, rather than in cost-plus, competitors alone or a round-number rise?

    Passes when Uses the willingness-to-pay and value evidence in the material to set each price, and says where the evidence is thin.

  3. Moves existing customers fairly

    Does the proposal say how existing customers move to the new pricing (who changes first, with what notice, and the value that justifies it)?

    Passes when A transition plan: new customers first or a set notice period for existing ones, tied to value delivered, with how churn will be watched.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 3 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Head of Product at Parsewell. Our CFO, Tom Reyes, wants every price up 30% in January, seats and enterprise contracts alike, citing the price test below. Our CEO has asked you for the alternative: a new pricing model, what it does to each kind of customer and to gross margin, how we get there (including the enterprise renewals), and what you make of the price test. No more than 1,300 words, with a table of the model and its effect by segment. All of our accounts are in the usage export. The files are attached.

What the model was given7 items: About Parsewell, current_pricing.md, usage_last_30_days.csv (every account), price_test_readout.md, enterprise_contracts.md, competitor_pricing.md, cfo_note.md
About ParsewellAn API that extracts structured data from documents (invoices, claims, bills of lading). Developers integrate it; customers are charged as described in the pricing file.
current_pricing.md5 lines · Download
# Current pricing

- Self-serve: $99 per seat per month. A seat is a developer or admin with access to the dashboard. Unlimited pages.
- Enterprise: annual contracts, priced per seat at a negotiated discount. Unlimited pages.
- Cost to serve: about $0.0012 per page (model inference and storage), and falling about 10% a quarter.
usage_last_30_days.csv (every account)account,segment,seats,pages_last_30d,revenue_last_30d_usd,cost_to_serve_last_30d_usd,gross_margin_usd Northwind Logistics,Enterprise (contract),12,4820000,3200,5784,-2584 Halden Insurance,Enterprise (contract),9,3960000,2600,4752,-2152 Arcadia Health,Enterprise (contract),15,3310000,4500,3972,528 Brightwater Bank,Enterprise (contract),10,2740000,3000,3288,-288 Copperfield Freight,Enterprise (contract),8,2380000,2400,2856,-456 Meridian Claims,Enterprise (contract),11,2150000,3300,2580,720 Mid-market A,Mid-market (self-serve),5,166000,495,199,296 Mid-market B,Mid-market (self-serve),8,291000,792,349,443 Mid-market C,Mid-market (self-serve),7,301000,693,361,332 Mid-market D,Mid-market (self-serve),6,288000,594,346,248 Mid-market E,Mid-market (self-serve),5,254000,495,305,190 Mid-market F,Mid-market (self-serve),4,235000,396,282,114 Mid-market G,Mid-market (self-serve),4,159000,396,191,205 Mid-market H,Mid-market (self-serve),8,330000,792,396,396 Mid-market I,Mid-market (self-serve),5,366000,495,439,56 Mid-market J,Mid-market (self-serve),5,234000,495,281,214 Mid-market K,Mid-market (self-serve),7,433000,693,520,173 Mid-market L,Mid-market (self-serve),7,134000,693,161,532 Mid-market M,Mid-market (self-serve),6,222000,594,266,328 Mid-market N,Mid-market (self-serve),6,196000,594,235,359 Mid-market O,Mid-market (self-serve),8,428000,792,514,278 Mid-market P,Mid-market (self-serve),3,251000,297,301,-4 Mid-market Q,Mid-market (self-serve),6,544000,594,653,-59 Mid-market R,Mid-market (self-serve),6,392000,594,470,124 Mid-market S,Mid-market (self-serve),8,350000,792,420,372 Mid-market T,Mid-market (self-serve),8,553000,792,664,128 Mid-market U,Mid-market (self-serve),8,192000,792,230,562 Mid-market V,Mid-market (self-serve),3,346000,297,415,-118 Mid-market W,Mid-market (self-serve),4,157000,396,188,208 Mid-market X,Mid-market (self-serve),5,145000,495,174,321 Startup 1,Startup (self-serve),3,33000,297,40,257 Startup 2,Startup (self-serve),3,40000,297,48,249 Startup 3,Startup (self-serve),1,2000,99,2,97 Startup 4,Startup (self-serve),3,21000,297,25,272 Startup 5,Startup (self-serve),2,25000,198,30,168 Startup 6,Startup (self-serve),2,10000,198,12,186 Startup 7,Startup (self-serve),1,19000,99,23,76 Startup 8,Startup (self-serve),2,29000,198,35,163 Startup 9,Startup (self-serve),1,11000,99,13,86 Startup 10,Startup (self-serve),2,14000,198,17,181 Startup 11,Startup (self-serve),1,9000,99,11,88 Startup 12,Startup (self-serve),2,16000,198,19,179 Startup 13,Startup (self-serve),3,32000,297,38,259 Startup 14,Startup (self-serve),3,15000,297,18,279 Startup 15,Startup (self-serve),1,35000,99,42,57 Startup 16,Startup (self-serve),1,35000,99,42,57 Startup 17,Startup (self-serve),3,27000,297,32,265 Startup 18,Startup (self-serve),1,26000,99,31,68 Startup 19,Startup (self-serve),3,26000,297,31,266 Startup 20,Startup (self-serve),2,14000,198,17,181 Startup 21,Startup (self-serve),3,20000,297,24,273 Startup 22,Startup (self-serve),3,16000,297,19,278 Startup 23,Startup (self-serve),3,39000,297,47,250 Startup 24,Startup (self-serve),1,9000,99,11,88 Startup 25,Startup (self-serve),1,16000,99,19,80 Startup 26,Startup (self-serve),1,10000,99,12,87 Startup 27,Startup (self-serve),3,27000,297,32,265 Startup 28,Startup (self-serve),2,39000,198,47,151 Startup 29,Startup (self-serve),2,18000,198,22,176 Startup 30,Startup (self-serve),1,15000,99,18,81 Startup 31,Startup (self-serve),3,11000,297,13,284 Startup 32,Startup (self-serve),1,40000,99,48,51 Startup 33,Startup (self-serve),2,32000,198,38,160 Startup 34,Startup (self-serve),2,39000,198,47,151 Startup 35,Startup (self-serve),1,15000,99,18,81 Startup 36,Startup (self-serve),3,12000,297,14,283 Startup 37,Startup (self-serve),1,21000,99,25,74 Startup 38,Startup (self-serve),3,19000,297,23,274 Startup 39,Startup (self-serve),2,38000,198,46,152 Startup 40,Startup (self-serve),2,32000,198,38,160 Startup 41,Startup (self-serve),2,5000,198,6,192 Startup 42,Startup (self-serve),2,18000,198,22,176 Startup 43,Startup (self-serve),2,35000,198,42,156 Startup 44,Startup (self-serve),2,31000,198,37,161 Startup 45,Startup (self-serve),3,21000,297,25,272 Startup 46,Startup (self-serve),2,7000,198,8,190 Startup 47,Startup (self-serve),2,30000,198,36,162 Startup 48,Startup (self-serve),3,3000,297,4,293 Startup 49,Startup (self-serve),2,28000,198,34,164 Startup 50,Startup (self-serve),1,8000,99,10,89
price_test_readout.md10 lines · Download
# Price test readout (Growth team, January)

Test: new self-serve sign-ups saw $129 per seat (B) or $99 (A), 8 to 21 December. Intended split 50/50.

| Arm | Pricing page visitors | Started paying | Conversion |
|---|---|---|---|
| A ($99) | 1,240 | 62 | 5.0% |
…
enterprise_contracts.md10 lines · Download
# Enterprise contracts

| Account | Annual value | Renews | Note |
|---|---|---|---|
| Northwind Logistics | $38,400 | Feb 2027 | Pages up 3x in a year; wants a 2-year renewal "at the same rate" |
| Halden Insurance | $31,200 | Jun 2027 | |
| Arcadia Health | $54,000 | Mar 2027 | Evaluating a competitor |
…
competitor_pricing.md5 lines · Download
# Competitor pricing (public pricing pages, September)

- DocuLens: $0.012 per page up to 1M pages a month, $0.008 above that; $500 a month platform fee.
- ExtractIQ: $0.015 per page, volume discounts from 500k pages; no seat charges.
- Formwise: per seat ($79), page limits per seat (50k), overage $0.01 per page.
cfo_note.md1 lines · Download
From Tom (CFO): "Gross margin was 41% a year ago and it's 24% now. Let's put every price up 30% in January, seats and contracts, and use the price test as cover. Customers expect increases."
What a strong answer doesThe answer key the graders mark against

Shows that the problem is the value metric, not the price level. Six enterprise accounts process about 71% of all pages (19.4M of 27.4M) on seat-based contracts worth $19,000 a month, costing about $23,200 to serve: they lose about $4,200 a month, while startups pay for seats they barely use (margin about 87%). Overall gross margin is about 24% ($43,100 revenue against $32,900 cost). A 30% rise on seats barely touches the loss-making volume (enterprise contracts would rise only at renewal) and lands hardest on small self-serve accounts. Proposes a hybrid: a platform fee plus a per-page price with volume tiers, priced against competitors (about $0.008 to $0.015 a page) and well above cost ($0.0012), and works out its effect by segment. Handles the enterprise renewals one by one (Copperfield in January, Northwind in February asking for the same rate on tripled volume, Arcadia in March evaluating a competitor) with usage-based terms or committed volume. Rejects the price test as evidence: the split is 1,240 to 870 when it should be 50/50 (a sample ratio mismatch), it ran over two holiday weeks, and with about 1,000 visitors an arm it couldn't detect the drop it claims isn't there. Plans a transition for existing self-serve customers and a way to measure it.

Critical failures (cap the score at 40)
  • Recommends the flat 30% rise on seats and contracts
  • Agrees to renew Northwind for two years at the same rate
Task

Staff-level · v1.1

The outputs

8 setups answered this brief.

Usable with a quick edit10 of 13 checks passed

The output is comprehensive, well-structured, and directly actionable; a product manager could use it with only minor wording edits.

Output

Recommendation: charge for document processing, not collaborators

Tom is right about the margin problem, but a blanket increase treats the symptom. Replace unlimited processing with a platform fee plus metered pages; make seats unlimited. Launch for new customers in January, migrate existing self-serve accounts after notice, and renegotiate enterprise contracts at renewal—not unilaterally.

The complete export shows $43,057 revenue, $32,903 processing cost and 23.6% gross margin. Enterprise accounts generate 71% of pages but 44% of revenue, losing $4,232 monthly. Startups generate $8,688 gross profit and are subsidizing heavy users.

Even with no churn and immediate contractual permission, Tom’s 30% increase produces only 41.2% overall margin and 5.9% enterprise margin. Northwind and Halden remain loss-making.

Proposed model and customer impact

A billable page is a successfully processed page; failures and internal retries are not charged. Provide usage dashboards, budget alerts and optional customer-controlled hard limits. Never silently interrupt production traffic.

The following is a fully migrated, unchanged-usage scenario, using reported account costs. It excludes churn, growth, transition concessions and future cost reductions.

SegmentProposed modelMonthly revenue: current → proposedGross margin: current → proposedCustomer effect
---------:---:---
Startups: 50 accountsSelf-serve: $99/account/month, including 25,000 pages; $0.004/additional page; unlimited seats$9,999 → $5,62286.9% → 76.7%Bills become $99–$159. 35 accounts pay less, 11 unchanged, 4 pay more. Aggregate spend falls 44%.
Mid-market: 24 accountsSame self-serve plan; enterprise available when annual commitment and enterprise features justify it$14,058 → $27,84440.5% → 70.0%Bills become $535–$2,211. 22 pay more, 2 less; changes range from −23% to +366%. Heavy users lose their unlimited-use subsidy.
Enterprise: 6 accounts$500/month platform + $0.003/page, with an annual take-or-pay page commitment, minimum 12M pages/year; excess pages at the same rate; unlimited seats, SSO, audit logs and contracted support$19,000 → $61,080−22.3% → 62.0%Monthly-equivalent bills become $6,950–$14,960: increases of 111%–376%. Major commercial renegotiations, not routine uplifts.
Total: 80 accounts$43,057 → $94,54623.6% → 65.2%Monthly gross profit rises from $10,154 to $61,643, before customer response.

Enterprise projections assume commitments do not exceed observed annualized consumption. Final commitments require longer usage histories and seasonality analysis.

This model deliberately sacrifices $4,377/month of startup revenue to lower adoption and collaboration friction. That is an investment to validate, not “free” growth. Conversely, mid-market V would move from $297 to $1,383; we should expect objections and possible churn.

The enterprise rate yields 60% incremental processing margin at today’s cost, before the platform fee. It remains well below the published competitor page rates, although September list prices do not establish negotiated prices or willingness to pay. If unit costs fall another 10%, unchanged-volume portfolio margin reaches approximately 68.7%; we should not spend those savings before they materialize.

These margins use the supplied cost definition. Finance must verify whether support and other delivery costs belong in COGS before adopting formal margin targets.

Migration and enterprise renewals

January: offer the model to new customers as a controlled rollout. Before billing, reconcile meters against infrastructure usage and invoices. Give existing self-serve accounts 90 days’ notice, two shadow invoices, and an account-specific comparison. Contact every materially affected mid-market account; offer usage optimization and narrowly approved, expiring credits rather than indefinite unlimited-use grandfathering.

Removing seat charges also requires workspace-level billing and anti-splitting controls for included allowances. Publish clear page-counting rules and a dispute process.

Enterprise: honor existing agreements through renewal unless customers voluntarily amend them. Confirm notice requirements immediately; if a deadline has passed, follow the contract. Use actual billed revenue as the baseline: the file’s description of discounted enterprise seats does not reconcile with observed revenue per seat.

For the 2027 renewal cycle:

RenewalAccountTarget monthly equivalent at current pagesNegotiation approach
---------:---
JanuaryCopperfield Freight$7,640Immediate executive outreach; first negotiated migration, subject to notice rights.
FebruaryNorthwind Logistics$14,960Reject two more years of unlimited processing “at the same rate.” Offer two-year per-page price certainty only with a committed volume and paid excess usage.
MarchArcadia Health$10,430Start retention work now: benchmark accuracy, reliability and switching costs against its competing offer.
JuneHalden Insurance$12,380Begin budget discussions well ahead of renewal; quantify its current subsidy and forecast demand.
SeptemberMeridian Claims$6,950Agree annual page commitment and seasonal pooling.
NovemberBrightwater Bank$8,720Allow procurement and security review lead time; document service commitments.

For difficult renewals, authorize a 90-day bridge at $500 plus $0.002/page, then the target rate. This preserves more than 40% processing margin at current usage while providing an explicit transition—not another unlimited contract. Finance approves exceptions and their gross-profit cost.

If existing self-serve accounts migrated while all enterprise contracts remained unchanged, portfolio margin would be only 37.3%. The 65.2% figure is a destination, not a January forecast. Build the operating plan from renewal dates, signed commitments, concessions and account-level retention scenarios. Be willing to lose persistently uneconomic volume, but do not confuse that with costless churn.

What the price test actually says

It does not justify a universal increase.

  • The intended 50/50 allocation failed: 1,240 versus 870 visitors gives a sample-ratio-mismatch statistic of approximately 64.9, with p < 10⁻¹⁴. Investigate assignment, exposure logging, exclusions, bots, acquisition channels and repeat visitors before causal interpretation.
  • “Not significant” does not mean equivalent. Even ignoring allocation concerns, conversion changed by approximately −0.17 percentage points, with an approximate 95% interval of −2.0 to +1.7 points—wide enough to include substantial harm.
  • The population and outcome are wrong for Tom’s conclusion: new self-serve sign-ups over two weeks tell us nothing directly about installed-base churn, enterprise renewals, seat reductions, usage cost or durable gross profit.
  • $129 is a 30.3% increase over $99, but the test measures neither realized revenue per visitor nor longer-term contribution.

Repair the experiment, then test the new model against current pricing with persistent account-level assignment. Predefine sample size, conversion guardrails and 90-day gross profit per eligible visitor; also track activation, retained usage, bill disputes and retention by segment. Use negotiated enterprise pilots—not this acquisition test—to validate renewal economics.

Grades and run details

Decision model 73 · LLM judge 93
Decision model checks
  • failUses the supplied evidence correctly45%
  • passAddresses the actual decision81%
  • partialRespects explicit constraints35%
  • passIdentifies material uncertainty81%
  • partialAvoids unsupported claims27%
  • passProduces the required deliverable66%
  • passFixes the value metric100%
  • passDoesn't trust the price test96%
  • passPlans the renewals100%
  • passEach tier for a different buyer86%
  • failGets the base of every number right76%
  • partialPriced on willingness to pay64%
  • passMoves existing customers fairly100%
Run
Run
#1
Time to output
1.8 min
Submitted
5 Oct 2026

Usable with a quick edit8 of 13 checks passed

The output is comprehensive and actionable; minor omissions like churn monitoring and a decision condition can be added with a quick edit without redoing the reasoning.

Output

To: CEO From: Head of Product Re: Alternative to the across-the-board 30% increase

Summary

Margin fell from 41% to 24% because pages are free inside the seat price and our cost is per page. Six enterprise accounts use 71% of all pages and pay 44% of revenue. Their combined gross margin is −22% (−$4.2k on $19.0k last month), and four of the six lose money. Startups run at 87% margin and mid-market at 41%.

A flat 30% doesn't fix this, for three reasons: - Contracts can't be repriced in January. All six renew in 2027, and I haven't seen any reopener clauses. - Enterprise stays near zero margin. Even at +30%, enterprise would reach only about 6% margin, and it would erode again as pages grow. Northwind's pages tripled in a year. - The price test doesn't support it (below).

I recommend pricing on pages, which is what drives our cost.

Proposed model and effect by segment

Figures are monthly, at last-30-day volumes and today's $0.0012 cost per page, with no churn.

SegmentNew pricingRevenue now → newWho pays whatGross margin now → new
Startup (50)$99/seat, now including 50k pages/seat (pooled). Overage $0.005/page.$10.0k → $10.0kNo change for any account; the heaviest uses 40k on one seat87% → 87%
Mid-market (24)$99/seat including 50k pages/seat (pooled). Overage $0.005/page.$14.1k → $19.3k (+37%)14 accounts unchanged. 10 pay $20 to $980 more; the three loss-makers (P, Q, V) rise 2–4x41% → 57%
Enterprise (6)Annual commit at $0.002/page, no seat charges, overage at the same rate, multi-year price lock for committed volume$19.0k → $38.7k (+104%)+30% (Meridian) to +205% (Halden); Northwind +201%−22% → 40%
Total$43.1k → $68.0k24% → 52%

Why these rates

  • Overage at $0.005: about 76% margin on the extra pages. It is below the cheapest competitor's per-page rate ($0.008).
  • Enterprise at $0.002: 40% margin today. If cost keeps falling 10% a quarter, it reaches about $0.0008 after four quarters, which puts this rate near 60% margin. That gives room to concede on rate later without losing money.
  • Competitor comparison: at Northwind's 4.8M pages, DocuLens list price is about $43k a month. We would charge $9.6k. Arcadia is evaluating a competitor, but at its 3.3M pages DocuLens lists near $31k against our $6.6k.
  • Enterprise seats: I'm dropping seat charges. Seats don't drive cost and mostly add negotiating friction.

Gross margin comparison

ScenarioRevenueGross margin
Today$43.1k23.6%
CFO plan, what January can actually do (self-serve +30%, contracts locked)$50.3k34.5%, and falling as enterprise pages grow
CFO plan, theoretical (every contract also +30%)$56.0k41.2%
Proposed model, at full run-rate$68.0k51.6%

The CFO plan assumes zero churn. Even on that assumption, its best case only gets back to last year's 41%.

Getting there

Self-serve (mid-market and startup): - Meter and display pooled page usage in the dashboard now. - Send mid-market a 60-day notice with a personal usage preview. Launch the new plan around April 1. - Charge overage at half rate for the first quarter, which softens the three biggest increases. - Startups see only a new allowance line, with no change in price.

Enterprise: Start before the renewal dates, and have legal check the contracts first. - Northwind (Feb 2027): Decline "the same rate". At 3x page growth again, its cost would be about $17k a month against $3.2k of revenue. Offer a two-year deal instead, with a price lock, a committed volume and a modest discount above 6M pages. That gives them the certainty they want. - Halden (Jun 2027): Same structure as Northwind, with an early-renewal offer. - Arcadia (Mar 2027): It has the smallest increase of the six accounts (+47%) and a competitor in play. Lead with value, and keep the rate flat to the competitor's list. - Copperfield (Jan 2027): First to renew, so the first live test of the model. Start talks in Q2 2026. - Meridian and Brightwater: Standard offer at renewal.

Northwind and Halden together lose about $4.7k a month, so losing them would improve gross margin dollars. We can hold the line. For the two losers the loss continues until renewal unless they take an early-renewal offer, so I'd offer a glide path (a year-one rate between $0.0015 and $0.002) in exchange for committing early. That keeps even the worst case above cost.

The price test

I wouldn't use it as cover, because it doesn't show what the readout says.

  1. The split is broken. A 50/50 split of 2,110 visitors should give about 1,055 per arm. We got 1,240 and 870, which is a 59/41 split. That is far too lopsided to be chance (p well below 0.001). Something in assignment or tracking failed, so the arms aren't comparable.
  2. "Not significant" is not "no effect". The difference is −0.17 points, with a 95% interval of about −2.0 to +1.7 points. $129 pays off only if conversion stays above about 3.8% (a drop of less than 23%), and the data can't rule out falling below that.
  3. It's the wrong test. It ran two weeks in December. It covers new sign-ups only and measures starting to pay, not retention or usage. It says nothing about existing customers, enterprise, or per-page pricing. Under my plan startups don't change price at all.

To test $129 properly, we'd need working randomisation and revenue per visitor plus 90-day retention as the measures. Detecting a one-point conversion change needs about 7,500 visitors per arm. I'd run that only if we want a startup price rise later.

Risks and assumptions

  • Figures assume static volumes, no churn, and today's unit cost. Enterprise page growth raises both revenue and cost under the new model, and the per-page rate protects margin.
  • Mid-market bill shock is the main self-serve risk. The three accounts with 2–4x increases are the ones most worth a personal call. They are also the three loss-makers (P, Q and V).
  • A 2–3x enterprise increase may cost us one or two accounts. I think that is acceptable, but it is a judgment call.
  • Full enterprise run-rate arrives only after the last renewal in November 2027. Savings this year come mostly from self-serve and any early renewals.
  • Not yet checked: contract terms (fair-use or reopener clauses) and whether billing can meter pooled pages by April.

Decision requested: approve the model for planning, and let me begin the Copperfield and Northwind conversations this quarter.

Grades and run details

Decision model 77 · LLM judge 79
Decision model checks
  • failUses the supplied evidence correctly13%
  • passAddresses the actual decision70%
  • passRespects explicit constraints38%
  • passIdentifies material uncertainty71%
  • partialAvoids unsupported claims41%
  • passProduces the required deliverable77%
  • passFixes the value metric100%
  • passDoesn't trust the price test100%
  • passPlans the renewals100%
  • passEach tier for a different buyer87%
  • failGets the base of every number right45%
  • partialPriced on willingness to pay33%
  • passMoves existing customers fairly83%
Run
Run
#1
API response time
2.1 min
Submitted
6 Oct 2026

Usable with a quick edit11 of 13 checks passed

The memo is complete, evidence-backed, and directly actionable by the CEO with only minor wording tweaks possible.

Output

Parsewell pricing: an alternative to a flat 30% increase

To: CEO | From: Head of Product

Recommendation

Tom has correctly identified the margin problem but proposes the wrong fix. We don't charge for pages, and pages are the only thing that costs us money. I recommend we keep seats for self-serve, add a page allowance with overage, and price enterprise per page. At today's volumes this takes gross margin from 24% to 42%. Startups pay nothing extra, and margin keeps rising as cost per page falls. A 30% increase reaches 41% on paper, but only if nobody churns and we could reprice contracts in January. We can't. And it leaves the cause in place.

Where the margin went

All figures come from the 30-day export (80 accounts, 27.4M pages, $43,057 revenue, $32,903 cost to serve).

  • Enterprise is underwater. It earns $0.00098 per page against a cost of $0.0012, a gross margin of −22%. Northwind and Halden alone lose $4,736 a month.
  • Mid-market makes 41%. Three accounts (P, Q, V) lose money because a few seats push heavy volume.
  • Startups make 87%. They are not the problem.

Seats don't track cost. Northwind's pages tripled in a year on the same contract. Cost per page falls about 34% a year (10% a quarter), but volume growth of that size outruns it. That is how margin fell from 41% to 24%.

The model

SegmentAcctsNew pricingRevenue/mo now → newGM now → new
Startup (self-serve)50$99/seat, 50k pages per seat included, $0.003/page above$9,999 → $9,999 (no account changes)87% → 87%
Mid-market (self-serve)24Same plan$14,058 → $17,214 (+22%); 14 accounts unchanged, 10 pay overage41% → 51%
Enterprise (annual)6$0.0015/page on a committed annual volume, overage at the same rate, unlimited seats, minimum = current contract value$19,000 → $29,115 (+53%)−22% → 20%
Total80$43,057 → $56,328 (+31%)24% → 42%

Why this shape:

  • Startups: no startup exceeds 50k pages per seat, so the change is invisible to them. That protects the funnel against Formwise at $79.
  • Mid-market: overage lands only on the accounts that cost us.
  • Enterprise: a single per-page rate with no volume discount, because our cost is linear in pages.

Margin over time: at $0.0015, enterprise margin is 20% at today's cost and about 48% by late 2027 if the cost trend holds. Unlike today, revenue grows with volume.

Competitive position: $0.0015 is deliberately conservative. DocuLens's public price for Northwind's volume would be about $43,000 a month, against $7,230 under this model. We stay the cheapest option for every mid-market account too. Account V (3 seats, 346k pages) would pay $885 with us, $2,197 at Formwise and about $4,650 at DocuLens.

Enterprise renewals

Contracts can't be repriced mid-term. A January increase on "contracts" would reach only Copperfield. We move each account at renewal.

AccountRenewsNow/moNew/moChangeApproach
CopperfieldJan 2027$2,400$3,570+49%First renewal; prove the motion and refine the talk track
NorthwindFeb 2027$3,200$7,230+126%Decline "same rate"; offer 2 years at $0.0015 then $0.0013
ArcadiaMar 2027$4,500$4,965+10%Retention case: hold $54k, cover up to 3.6M pages/mo
HaldenJun 2027$2,600$5,940+128%Same as Northwind
MeridianSep 2027$3,300$3,3000%Minimum applies; framing benefit only
BrightwaterNov 2027$3,000$4,110+37%Standard

Northwind. "Same rate for two years" means unlimited pages for $38,400 a year. If their growth continues, they reach roughly 14M pages a month. Even after two more years of cost decline, that would lose us about $100k a year.

  • Offer: two years at $0.0015, stepping down to $0.0013 in year two. This gives them the price certainty they want while we keep a margin on growth.
  • Walk-away point: we should be willing to lose them. Northwind and Halden leaving would cut $5,800 a month of revenue but raise gross profit by $4,736 a month. That fact sets our walk-away point; it isn't a threat to put in front of them.

Arcadia already pays $0.00136 per page and is looking at a competitor.

  • Offer: hold them at $54k for 12 months covering up to 3.6M pages a month, with $0.0015 above that.
  • Discovery: competitors' public prices are far above ours, so we should find out what is really driving the evaluation before assuming it's price.

Phasing. Full enterprise effect arrives with the November renewal. Margin climbs through 2027 rather than jumping in January.

Self-serve rollout

  1. January: new sign-ups get the new plan. Existing accounts get an in-dashboard usage meter showing their allowance.
  2. Notice: existing accounts get 60 days' notice, with overage billing from April.
  3. Personal outreach to five accounts: I, P, Q, T and V see increases of 58% to 198%. We contact each one and offer a committed annual page pack at $0.002.
  4. Worst case: if all five churned, we lose $2,433 a month of new gross profit. Blended margin would still be about 41%.

The price test

The test doesn't support a 30% increase, for the following reasons.

  1. The split is broken. It was meant to be 50/50 but came out 1,240 vs 870 visitors (59/41). The chance of that happening randomly is effectively zero (χ² ≈ 65). Something in assignment or tracking failed, possibly losing B-arm visitors who bounced. Until that is explained, neither arm's numbers can be trusted.
  2. "Not significant" doesn't mean "no effect." With 104 conversions, the 95% confidence interval on the difference runs from −2.0 to +1.7 points. The data are consistent with a 40% drop in conversion.
  3. The timing and measure are weak. It ran for two holiday weeks and measured first payment only, not retention or seat growth.
  4. It covered the wrong customers for this decision. It tested new self-serve sign-ups at $129. It says nothing about existing customers or enterprise contracts. Enterprise buyers will ask for the evidence, and citing this test for their increase would hurt our credibility at exactly the renewals that matter.

At face value, B shows higher revenue per visitor ($6.23 vs $4.95). That makes the test worth rerunning, not acting on. A valid rerun needs:

  • a fixed split;
  • four or more weeks;
  • 60-day paid retention as the success measure;
  • the new allowance plan as one of the arms.

Compared with a flat 30%

Flat +30% (as proposed)Flat +30% (what January allows)This model
Gross margin today41%36%42%
Northwind / HaldenStill lose $1,624 / $1,372 a monthUnchanged losses20% margin
As pages growMargin erodes againMargin erodes againMargin holds and rises as costs fall
Startups (87% margin, most price-sensitive)+30%+30%Unchanged
Churn assumedNoneNoneDownside quantified above

Decisions needed

  1. Approve the model and the $0.0015 enterprise rate.
  2. Authorize the walk-away position on Northwind and Halden.
  3. Ask Growth to diagnose the split failure before any test result is cited.

I'd like Tom in the room for the first two. His margin target is right, and this model reaches it faster and keeps it.

Grades and run details

Decision model 77 · LLM judge 100
Decision model checks
  • failUses the supplied evidence correctly26%
  • passAddresses the actual decision98%
  • passRespects explicit constraints28%
  • passIdentifies material uncertainty87%
  • partialAvoids unsupported claims30%
  • passProduces the required deliverable76%
  • passFixes the value metric90%
  • passDoesn't trust the price test100%
  • passPlans the renewals100%
  • passEach tier for a different buyer90%
  • failGets the base of every number right71%
  • partialPriced on willingness to pay28%
  • passMoves existing customers fairly100%
Run
Run
#1
Time to output
2.9 min
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 3

Addresses the actual decisionMixedMixedRight
GPT-6 Astra · ChatGPT

The output commits to a new pricing model but does not state what specific result or condition would change that recommendation.

Sonnet 5.5 · API

The output commits to a recommendation but does not state what specific result or condition would change that call, only noting it is a judgment call.

Opus 5.5 · Claude

Commits early to a new hybrid pricing model, addresses the CEO's request, and states what would change the call (Tom's margin target, walk-away authorization).

Identifies material uncertaintyRightMixedRight
GPT-6 Astra · ChatGPT

It names unknowns (seasonality, cost definitions, competitor willingness to pay, churn) and says how to resolve them (longer histories, finance verification, retention scenarios).

Sonnet 5.5 · API

It names some unknowns (contract terms, billing capability) but does not bound them or specify what would change the decision, and does not say how churn risk would be resolved beyond a judgment call.

Opus 5.5 · Claude

Identifies the broken test split, churn risk from five mid-market accounts, Arcadia's true evaluation driver, and bounds the downside.

Moves existing customers fairlyRightMixedRight
GPT-6 Astra · ChatGPT

Transition plan: new customers first, 90-day notice with shadow invoices for self-serve, enterprise at renewal, with churn monitoring and credits for affected accounts.

Sonnet 5.5 · API

The transition plan includes notice periods and value justification but does not specify how churn will be monitored or watched.

Opus 5.5 · Claude

Transition plan: new sign-ups in January, 60-day notice for existing, personal outreach to five heavy users, and churn downside quantified.

All mixed 2

Uses the supplied evidence correctlyMixedMixedMixed
GPT-6 Astra · ChatGPT

All claims about the current situation are taken directly from the supplied data or derived by correct arithmetic.

Sonnet 5.5 · API

All factual claims about the current situation are directly supported by the supplied context or arithmetic from it.

Opus 5.5 · Claude

All factual claims about the current situation are directly supported by the supplied context or arithmetic from it.

Gets the base of every number rightMixedMixedMixed
GPT-6 Astra · ChatGPT

All derived figures (margins, percentages, price increase) check out against the supplied data and correct arithmetic.

Sonnet 5.5 · API

All derived figures (margins, revenue changes, percentages) check out against the supplied data with correct denominators.

Opus 5.5 · Claude

All derived figures (margins, per-page rates, competitor comparisons) check out against the supplied data, and bases are stated where ambiguous.

All got right 8

Respects explicit constraintsRightRightRight
GPT-6 Astra · ChatGPT

The output is well under 1,300 words, includes the required table, and addresses the CEO's request for an alternative.

Sonnet 5.5 · API

The output is a memo to the CEO, includes the required table, and is well within the 1,300-word limit.

Opus 5.5 · Claude

Memo to CEO, under 1,300 words, includes the required table of model and effect by segment, and respects all explicit constraints.

Avoids unsupported claimsRightRightRight
GPT-6 Astra · ChatGPT

Interpretations and forecasts are clearly labeled as scenarios, assumptions, or the author's view, not as established fact.

Sonnet 5.5 · API

Interpretations like the cause of margin decline are directly supported by the cost and pricing structure; no unsupported claims are presented as fact.

Opus 5.5 · Claude

Causal claims are grounded in data; forecasts are clearly projections; the test split failure is labelled as 'possibly' a tracking issue.

Produces the required deliverableRightRightRight
GPT-6 Astra · ChatGPT

The memo provides a complete new pricing model, segment effects, migration plan, enterprise renewals, and price test critique, usable by the CEO.

Sonnet 5.5 · API

The memo is complete, addressed to the CEO, includes the required table and all requested elements, and is actionable.

Opus 5.5 · Claude

Complete memo with model, segment effects, gross margin impact, renewal plan, and price test assessment; usable as-is.

Fixes the value metricRightRightRight
GPT-6 Astra · ChatGPT

It computes enterprise share of pages (71%) and their negative margin, and proposes usage-based pricing with per-page tiers.

Sonnet 5.5 · API

It computes the enterprise share of pages (71%) and their negative margin, and proposes usage-based pricing with per-page tiers.

Opus 5.5 · Claude

Computes enterprise share of pages and negative margin, then proposes usage-based pricing with per-page tiers for enterprise and page allowance for self-serve.

Doesn't trust the price testRightRightRight
GPT-6 Astra · ChatGPT

It names the sample ratio mismatch and the wide confidence interval, and does not rest the decision on the test.

Sonnet 5.5 · API

It identifies the uneven split (59/41), the holiday timing, and the insufficient sample size, and does not rely on the test.

Opus 5.5 · Claude

Names the 59/41 split mismatch, holiday timing, and small sample; does not rest the decision on the test.

Plans the renewalsRightRightRight
GPT-6 Astra · ChatGPT

It gives a specific renewal approach for each near-term enterprise contract, using their dates and notes (e.g., rejecting Northwind's same-rate request, starting retention for Arcadia).

Sonnet 5.5 · API

It gives a specific approach for each near-term enterprise renewal, using their dates and notes (Copperfield, Northwind, Arcadia, etc.).

Opus 5.5 · Claude

Addresses each enterprise renewal with its date and note, including Copperfield in January, Northwind's tripled volume, and Arcadia's competitor evaluation.

Each tier for a different buyerRightRightRight
GPT-6 Astra · ChatGPT

Self-serve plan targets startups and mid-market with low entry and unlimited seats; enterprise plan targets high-volume buyers with a platform fee, lower per-page rate, and enterprise features.

Sonnet 5.5 · API

Each tier (startup, mid-market, enterprise) is built for a distinct buyer with a different value metric and pricing structure.

Opus 5.5 · Claude

Each tier names its buyer (startups, mid-market, enterprise) and is priced on distinct value: startups protected, mid-market overage on heavy users, enterprise per-page with unlimited seats.

Priced on willingness to payRightRightRight
GPT-6 Astra · ChatGPT

Prices are set below competitor page rates and above cost, with explicit acknowledgment that competitor list prices may not reflect willingness to pay.

Sonnet 5.5 · API

Prices are grounded in competitor rates and value delivered, with margin considerations, not just cost-plus or round numbers.

Opus 5.5 · Claude

Prices are grounded in competitor comparisons showing significant headroom, and acknowledges thin evidence for Arcadia by recommending discovery.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 81% of 68 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Opus 5.5withClaude76.0100.02None
2GPT-6 AstrawithChatGPT74.096.42None
3Sonnet 5.5withAPI76.089.32None
4GPT-6.1 SolwithAPI76.088.72None
5Haiku 5.5withAPI78.076.92None
6GPT-6 LunawithAPI74.064.02None
7Gemini 3.8 FlashwithAPI62.044.22None
8Gemini 3.5 Flash-LitewithGemini45.840.72None

What you get for the cost

Sonnet 5.5 · API scores within 5 points of Opus 5.5 · Claude at half the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreOpus 5.5 · Claude: task score 89, 122× the cheapestGPT-6 Astra · ChatGPT: task score 86, 108× the cheapestSonnet 5.5 · API: task score 84, 61× the cheapestGPT-6.1 Sol · API: task score 81, 20× the cheapestHaiku 5.5 · API: task score 76, 2.9× the cheapestGPT-6 Luna · API: task score 68, the cheapestGemini 3.8 Flash · API: task score 51, 9.1× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 42, 2.3× the cheapestOpus 5.5 · ClaudeGPT-6 Astra · ChatGPTSonnet 5.5 · APIGPT-6.1 Sol · APIHaiku 5.5 · APIGPT-6 Luna · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and cost
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API681.0×
Gemini 3.5 Flash-Lite · Gemini422.3×
Haiku 5.5 · API762.9×
Gemini 3.8 Flash · API519.1×
GPT-6.1 Sol · API8120×
Sonnet 5.5 · API8461×
GPT-6 Astra · ChatGPT86108×
Opus 5.5 · Claude89122×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Setting what you charge, for what, and to whom.

Why it matters

Most products are priced once and never revisited, on a metric that doesn't track value. The fix is rarely a flat rise.

What good looks like

  • Tiers built for distinct buyers
  • A value metric that tracks what customers get
  • Grounded in willingness to pay and cost to serve
  • A plan for existing customers

Deliberately not measured

  • Billing implementation
  • Legal terms
Capability tested

Pricing judgement

The failure we’re looking for

A flat price rise on the wrong value metric

Grading

Decision model and LLM judge, calibrated against a blind PM review