Tasks / Strategy

Price and package a product

Can the model price on the value each kind of customer gets, using real usage and willingness-to-pay data, and get existing customers there?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 56% were usable with at most a quick edit.

Reliably right

  1. Doesn't trust the price test100% pass
    Names the 59/41 split mismatch, holiday timing, and small sample; does not rest the decision on the test.
    Opus 5.5 · Claude · Unlimited pages, falling margins
  2. Plans the renewals100% pass
    Addresses each enterprise renewal with its date and note, including Copperfield in January, Northwind's tripled volume, and Arcadia's competitor evaluation.
    Opus 5.5 · Claude · Unlimited pages, falling margins
  3. Each tier for a different buyer94% pass
    Each tier names its buyer (startups, mid-market, enterprise) and is priced on distinct value: startups protected, mid-market overage on heavy users, enterprise per-page with unlimited seats.
    Opus 5.5 · Claude · Unlimited pages, falling margins

Where it slips

  1. Uses the supplied evidence correctly34% pass
    Total MRR is stated as $11,670 (actual $11,820), and median 'getting expensive' for small agencies is given as about $37 (actual $33) and for agencies as about $55 (actual $51).
    Haiku 5.5 · API · Fifteen dollars for everyone
  2. Gets the base of every number right41% pass
    MRR total and two segment‑level median getting‑expensive figures are incorrectly computed from the supplied data.
    Haiku 5.5 · API · Fifteen dollars for everyone
  3. Avoids unsupported claims52% pass
    Median getting‑expensive values and total MRR are presented as fact but are not supported by the data; these are facts about the current situation, not labelled as assumptions.
    Haiku 5.5 · API · Fifteen dollars for everyone

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Each tier for a different buyer

    Is each plan built for a distinct kind of buyer, with what that buyer values, rather than more-of-the-same at a higher price?

    Passes when Every tier names its buyer and the value it's priced on, and includes what that buyer needs and the lower tier's buyer doesn't.

  2. Priced on willingness to pay

    Are the proposed prices grounded in evidence of what each kind of customer would pay and the value they get, rather than in cost-plus, competitors alone or a round-number rise?

    Passes when Uses the willingness-to-pay and value evidence in the material to set each price, and says where the evidence is thin.

  3. Moves existing customers fairly

    Does the proposal say how existing customers move to the new pricing (who changes first, with what notice, and the value that justifies it)?

    Passes when A transition plan: new customers first or a set notice period for existing ones, tied to value delivered, with how churn will be watched.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 3 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Head of Product at Parsewell. Our CFO, Tom Reyes, wants every price up 30% in January, seats and enterprise contracts alike, citing the price test below. Our CEO has asked you for the alternative: a new pricing model, what it does to each kind of customer and to gross margin, how we get there (including the enterprise renewals), and what you make of the price test. No more than 1,300 words, with a table of the model and its effect by segment. All of our accounts are in the usage export. The files are attached.

What the model was given7 items: About Parsewell, current_pricing.md, usage_last_30_days.csv (every account), price_test_readout.md, enterprise_contracts.md, competitor_pricing.md, cfo_note.md
About ParsewellAn API that extracts structured data from documents (invoices, claims, bills of lading). Developers integrate it; customers are charged as described in the pricing file.
current_pricing.md5 lines · Download
# Current pricing

- Self-serve: $99 per seat per month. A seat is a developer or admin with access to the dashboard. Unlimited pages.
- Enterprise: annual contracts, priced per seat at a negotiated discount. Unlimited pages.
- Cost to serve: about $0.0012 per page (model inference and storage), and falling about 10% a quarter.
usage_last_30_days.csv (every account)account,segment,seats,pages_last_30d,revenue_last_30d_usd,cost_to_serve_last_30d_usd,gross_margin_usd Northwind Logistics,Enterprise (contract),12,4820000,3200,5784,-2584 Halden Insurance,Enterprise (contract),9,3960000,2600,4752,-2152 Arcadia Health,Enterprise (contract),15,3310000,4500,3972,528 Brightwater Bank,Enterprise (contract),10,2740000,3000,3288,-288 Copperfield Freight,Enterprise (contract),8,2380000,2400,2856,-456 Meridian Claims,Enterprise (contract),11,2150000,3300,2580,720 Mid-market A,Mid-market (self-serve),5,166000,495,199,296 Mid-market B,Mid-market (self-serve),8,291000,792,349,443 Mid-market C,Mid-market (self-serve),7,301000,693,361,332 Mid-market D,Mid-market (self-serve),6,288000,594,346,248 Mid-market E,Mid-market (self-serve),5,254000,495,305,190 Mid-market F,Mid-market (self-serve),4,235000,396,282,114 Mid-market G,Mid-market (self-serve),4,159000,396,191,205 Mid-market H,Mid-market (self-serve),8,330000,792,396,396 Mid-market I,Mid-market (self-serve),5,366000,495,439,56 Mid-market J,Mid-market (self-serve),5,234000,495,281,214 Mid-market K,Mid-market (self-serve),7,433000,693,520,173 Mid-market L,Mid-market (self-serve),7,134000,693,161,532 Mid-market M,Mid-market (self-serve),6,222000,594,266,328 Mid-market N,Mid-market (self-serve),6,196000,594,235,359 Mid-market O,Mid-market (self-serve),8,428000,792,514,278 Mid-market P,Mid-market (self-serve),3,251000,297,301,-4 Mid-market Q,Mid-market (self-serve),6,544000,594,653,-59 Mid-market R,Mid-market (self-serve),6,392000,594,470,124 Mid-market S,Mid-market (self-serve),8,350000,792,420,372 Mid-market T,Mid-market (self-serve),8,553000,792,664,128 Mid-market U,Mid-market (self-serve),8,192000,792,230,562 Mid-market V,Mid-market (self-serve),3,346000,297,415,-118 Mid-market W,Mid-market (self-serve),4,157000,396,188,208 Mid-market X,Mid-market (self-serve),5,145000,495,174,321 Startup 1,Startup (self-serve),3,33000,297,40,257 Startup 2,Startup (self-serve),3,40000,297,48,249 Startup 3,Startup (self-serve),1,2000,99,2,97 Startup 4,Startup (self-serve),3,21000,297,25,272 Startup 5,Startup (self-serve),2,25000,198,30,168 Startup 6,Startup (self-serve),2,10000,198,12,186 Startup 7,Startup (self-serve),1,19000,99,23,76 Startup 8,Startup (self-serve),2,29000,198,35,163 Startup 9,Startup (self-serve),1,11000,99,13,86 Startup 10,Startup (self-serve),2,14000,198,17,181 Startup 11,Startup (self-serve),1,9000,99,11,88 Startup 12,Startup (self-serve),2,16000,198,19,179 Startup 13,Startup (self-serve),3,32000,297,38,259 Startup 14,Startup (self-serve),3,15000,297,18,279 Startup 15,Startup (self-serve),1,35000,99,42,57 Startup 16,Startup (self-serve),1,35000,99,42,57 Startup 17,Startup (self-serve),3,27000,297,32,265 Startup 18,Startup (self-serve),1,26000,99,31,68 Startup 19,Startup (self-serve),3,26000,297,31,266 Startup 20,Startup (self-serve),2,14000,198,17,181 Startup 21,Startup (self-serve),3,20000,297,24,273 Startup 22,Startup (self-serve),3,16000,297,19,278 Startup 23,Startup (self-serve),3,39000,297,47,250 Startup 24,Startup (self-serve),1,9000,99,11,88 Startup 25,Startup (self-serve),1,16000,99,19,80 Startup 26,Startup (self-serve),1,10000,99,12,87 Startup 27,Startup (self-serve),3,27000,297,32,265 Startup 28,Startup (self-serve),2,39000,198,47,151 Startup 29,Startup (self-serve),2,18000,198,22,176 Startup 30,Startup (self-serve),1,15000,99,18,81 Startup 31,Startup (self-serve),3,11000,297,13,284 Startup 32,Startup (self-serve),1,40000,99,48,51 Startup 33,Startup (self-serve),2,32000,198,38,160 Startup 34,Startup (self-serve),2,39000,198,47,151 Startup 35,Startup (self-serve),1,15000,99,18,81 Startup 36,Startup (self-serve),3,12000,297,14,283 Startup 37,Startup (self-serve),1,21000,99,25,74 Startup 38,Startup (self-serve),3,19000,297,23,274 Startup 39,Startup (self-serve),2,38000,198,46,152 Startup 40,Startup (self-serve),2,32000,198,38,160 Startup 41,Startup (self-serve),2,5000,198,6,192 Startup 42,Startup (self-serve),2,18000,198,22,176 Startup 43,Startup (self-serve),2,35000,198,42,156 Startup 44,Startup (self-serve),2,31000,198,37,161 Startup 45,Startup (self-serve),3,21000,297,25,272 Startup 46,Startup (self-serve),2,7000,198,8,190 Startup 47,Startup (self-serve),2,30000,198,36,162 Startup 48,Startup (self-serve),3,3000,297,4,293 Startup 49,Startup (self-serve),2,28000,198,34,164 Startup 50,Startup (self-serve),1,8000,99,10,89
price_test_readout.md10 lines · Download
# Price test readout (Growth team, January)

Test: new self-serve sign-ups saw $129 per seat (B) or $99 (A), 8 to 21 December. Intended split 50/50.

| Arm | Pricing page visitors | Started paying | Conversion |
|---|---|---|---|
| A ($99) | 1,240 | 62 | 5.0% |
…
enterprise_contracts.md10 lines · Download
# Enterprise contracts

| Account | Annual value | Renews | Note |
|---|---|---|---|
| Northwind Logistics | $38,400 | Feb 2027 | Pages up 3x in a year; wants a 2-year renewal "at the same rate" |
| Halden Insurance | $31,200 | Jun 2027 | |
| Arcadia Health | $54,000 | Mar 2027 | Evaluating a competitor |
…
competitor_pricing.md5 lines · Download
# Competitor pricing (public pricing pages, September)

- DocuLens: $0.012 per page up to 1M pages a month, $0.008 above that; $500 a month platform fee.
- ExtractIQ: $0.015 per page, volume discounts from 500k pages; no seat charges.
- Formwise: per seat ($79), page limits per seat (50k), overage $0.01 per page.
cfo_note.md1 lines · Download
From Tom (CFO): "Gross margin was 41% a year ago and it's 24% now. Let's put every price up 30% in January, seats and contracts, and use the price test as cover. Customers expect increases."
What a strong answer doesThe answer key the graders mark against

Shows that the problem is the value metric, not the price level. Six enterprise accounts process about 71% of all pages (19.4M of 27.4M) on seat-based contracts worth $19,000 a month, costing about $23,200 to serve: they lose about $4,200 a month, while startups pay for seats they barely use (margin about 87%). Overall gross margin is about 24% ($43,100 revenue against $32,900 cost). A 30% rise on seats barely touches the loss-making volume (enterprise contracts would rise only at renewal) and lands hardest on small self-serve accounts. Proposes a hybrid: a platform fee plus a per-page price with volume tiers, priced against competitors (about $0.008 to $0.015 a page) and well above cost ($0.0012), and works out its effect by segment. Handles the enterprise renewals one by one (Copperfield in January, Northwind in February asking for the same rate on tripled volume, Arcadia in March evaluating a competitor) with usage-based terms or committed volume. Rejects the price test as evidence: the split is 1,240 to 870 when it should be 50/50 (a sample ratio mismatch), it ran over two holiday weeks, and with about 1,000 visitors an arm it couldn't detect the drop it claims isn't there. Plans a transition for existing self-serve customers and a way to measure it.

Critical failures (cap the score at 40)
  • Recommends the flat 30% rise on seats and contracts
  • Agrees to renew Northwind for two years at the same rate
Task

Staff-level · v1.1

The outputs

8 setups answered this brief.

Usable with a quick edit11 of 13 checks passed

The output is a complete, evidence-backed memo that a PM could finalize with light edits, as all reasoning is clear and no fact-checking or structural rework is needed.

Output

To: CEO From: Head of Product Re: Alternative to a 30% January increase

Recommendation

Don't apply a flat 30%. Move pricing from seats with unlimited pages to a model that charges for pages. Self-serve keeps seats but adds an included page allowance and a per-page overage. Enterprise moves to a committed per-page rate at renewal. At current usage, this takes gross margin from 24% to about 53%.

Tom's increase would not get us to 41%. That figure assumes all contracts reprice in January, and only one enterprise contract renews then. Self-serve-only in January gives about 34%.

What the data says

  • The margin problem is enterprise. Six contracts account for 71% of pages (19.4M of 27.4M) and 71% of serving cost ($23.2k of $32.9k a month). They pay $19.0k, so they lose $4.2k a month (−22%). Four of six are negative.
  • Enterprise pays too little per page. Effective rates run $0.0007 to $0.0015 per page against a serving cost of $0.0012. Seat pricing lets page volume grow without any increase in what we charge. Northwind's pages tripled in a year and its bill didn't move.
  • Self-serve is healthy. It runs at 60% margin. Startups are at 87%, and mid-market is at 40%, dragged down by heavy accounts. Q and P are negative.
  • We are far below market. DocuLens lists at $0.012 per page up to 1M and $0.008 above, plus a $500 platform fee. At that list price, Northwind's 4.8M pages would cost about $43k a month, against our $3.2k. Enterprise discounts narrow this gap, but not by much.

Proposed model

  • Self-serve: $99 per seat per month, including 25,000 pages per seat. $0.002 per page above that. Seats stay because they are simple and match Formwise. Our overage rate is one-fifth of Formwise's $0.01.
  • Enterprise: a committed monthly page volume at $0.002 per page, with overage at the same rate. No seat fee. Offered on two-year terms with a price lock.
  • Floor: $0.0017 per page. Anything below it needs CEO sign-off.

Because serving cost is falling about 10% a quarter, a locked rate becomes more profitable each quarter. If that holds, enterprise margin reaches about 60% within a year at current volume.

Effect by segment

Monthly run-rate at current usage, before churn, at $0.0012 per page.

SegmentAccounts (seats / pages per month)TodayProposedRevenue per monthGross margin
Startups (self-serve)50 (101 / 1.09M)$99/seat, unlimited pages$99/seat incl. 25k pages; $0.002 above$10.0k → $10.1k (+1%)87% → 87%
Mid-market (self-serve)24 (142 / 6.97M)$99/seat, unlimited pages$99/seat incl. 25k pages; $0.002 above$14.1k → $21.0k (+49%)40% → 60%
Enterprise6 (65 / 19.4M)Negotiated per seat, unlimited pages$0.002 per committed page, no seat fee$19.0k → $38.7k (+104%)−22% → 40%
Total80$43.1k → $69.8k (+62%)24% → 53%

Gross margin dollars rise from $10.2k to $36.9k a month. Startups barely change because most use less than 25k pages per seat. Mid-market carries the most self-serve risk: a few heavy accounts see large increases, such as Q ($594 to $1,382) and T ($792 to $1,498).

Getting there

Self-serve. Announce in January and take effect February 1 with 30 days' notice. New sign-ups get the new model immediately. For existing heavy mid-market accounts, cap the increase at 50% for the first three months. Offer annual prepay as an option.

Enterprise, by renewal date. No contract is repriced mid-term.

  • Copperfield (January 2027): $0.002 per page takes it from $2.4k to about $4.8k a month. This is the first test of the model.
  • Northwind (February 2027): Its ask is "two years at the same rate." At the same dollar amount, the effective rate would fall to about $0.0007 per page while volume triples. Counter with a two-year term, a committed 4.8M pages a month, $0.0017 in year one, and $0.002 in year two. That is about $8.2k a month in year one. Trade the rate for term and commitment. If they refuse to go below $0.0017, let them walk. They lose us $2.6k a month today, and losing them would improve gross margin dollars.
  • Arcadia (March 2027): It is evaluating a competitor and is the only enterprise account with positive margin ($528 a month). Offer $0.002, benchmark against ExtractIQ at their volume, and concede only down to the floor and only on a multi-year deal.
  • Halden (June), Meridian (September), Brightwater (November): standard $0.002.

The price test

The test's B arm was $129 against $99, which is a 30% increase. That makes it the most relevant evidence we have, and it doesn't support the January increase.

  • The result is inconclusive. Conversion was 5.0% against 4.8%. The 95% interval on the difference is roughly −1.7 to +2.1 points. That interval can't rule out losing a third of conversion. "Not significant" does not mean "no effect."
  • The assignment looks broken. The arms received 1,240 and 870 visitors against a 50/50 design. That is a sample-ratio mismatch (chi-square ≈ 65). Until someone explains it, we can't trust either arm.
  • Its scope is narrow. It ran two weeks over the holidays, measured new sign-ups only, and counted first payments only. It says nothing about existing customers, seat expansion, churn, or heavy page users.

We shouldn't cite it as cover. Rerun it properly: a six-week randomized 50/50 test of the new model on new sign-ups, plus a notice-based read on existing self-serve accounts.

Why not 30% across the board

  • It doesn't fix enterprise. A 30% increase moves enterprise from −22% to about 6% margin. Even then, it can't be applied mid-term.
  • Seats don't track cost. Serving cost is about $13 per seat for startups and about $480 per seat for Northwind. A per-seat increase raises both sides equally and leaves the mismatch in place.
  • It taxes the wrong customers. Light users absorb the same percentage increase as heavy users, while the heavy users are the ones we under-recover on.

Risks

  • Enterprise pushback. A roughly 2x jump at renewal is the real risk. Expect one or two losses. The two-year lock and staggered renewal calendar limit the damage.
  • Cost decline stalls. If serving cost stops falling, the $0.0017 floor needs review each quarter.
  • Usage shifts. Heavy self-serve users may trim usage to avoid overage. That is acceptable given the margin, but we should watch conversion closely.

Decisions requested

  1. Reject the flat 30% increase.
  2. Approve the self-serve model effective February 1, with a 90-day cap for heavy mid-market accounts.
  3. Approve the enterprise per-page rate card, with a $0.0017 floor and CEO sign-off required below it.
  4. Rerun the price test properly before any new-signup price change.

Grades and run details

Decision model 77 · LLM judge 100
Decision model checks
  • failUses the supplied evidence correctly16%
  • passAddresses the actual decision97%
  • passRespects explicit constraints38%
  • passIdentifies material uncertainty85%
  • partialAvoids unsupported claims37%
  • passProduces the required deliverable74%
  • passFixes the value metric100%
  • passDoesn't trust the price test100%
  • passPlans the renewals100%
  • passEach tier for a different buyer85%
  • failGets the base of every number right59%
  • partialPriced on willingness to pay34%
  • passMoves existing customers fairly95%
Run
Run
#1
API response time
1.9 min
Submitted
8 Oct 2026

Usable with a quick edit10 of 13 checks passed

The output is well-structured, evidence-based, and covers all required elements. A product manager could use it with minor wording tweaks, without rechecking data or redoing analysis.

Output

Recommendation to the CEO

Reject the blanket January increase. Replace unlimited-page seat pricing with a platform fee plus metered pages, and migrate enterprise accounts at contractual renewal. Validate the proposed rates before broad rollout; the December test does not establish willingness to pay for this model.

The problem is the pricing metric, not simply the price level. Dashboard seats do not track extraction value or cost. Six enterprise accounts generate 19.36 million pages—71% of all usage—but lose $4,232 a month at today’s costs. Meanwhile, small startups subsidize heavy users.

Across all 80 accounts, current monthly revenue is $43,057, cost to serve is $32,903, and gross margin is 23.6%, consistent with Tom’s reported 24%.

Proposed model and customer impact

Include unlimited dashboard users. Charge for successfully processed pages, with clear definitions for multipage documents, retries and failures. The calculations below assume exported pages are billable; validate that assumption before launch.

SegmentProposed monthly pricingEffect at current usageMonthly revenue: current → proposedGross margin: current → proposed
------------:---:
Startup: 50 accounts$99 platform fee, including 25,000 pages; $0.006/page thereafterBills become $99–$189. 35 pay less, 11 unchanged, 4 pay more. Aggregate spend falls 40%.$9,999 → $5,95886.9% → 78.0%
Mid-market: 24 accountsSame self-serve modelAll pay more: approximately 9%–582%. Bills become $753–$3,267. Aggregate spend rises 189%; high-volume, low-seat accounts see the largest corrections.$14,058 → $40,57840.5% → 79.4%
Enterprise: 6 accounts$500 platform fee + $0.004/page, with annual page commitments and quarterly true-ups; negotiated SLA/security termsFull-rate bills become $9,100–$19,780 monthly equivalents, 176%–528% above current contracts. Transition discounts below soften—not eliminate—the reset.$19,000 → $80,440−22.3% → 71.1%
Total, fully migratedNo seat charges; no unlimited-page promisesRevenue follows extraction volume rather than dashboard headcount.$43,057 → $126,97623.6% → 74.1%

These are static, fully migrated scenarios, not forecasts: they assume unchanged usage, no churn and today’s exported costs. Monthly gross profit rises from $10,154 to $94,073. A further 10% cost reduction would raise proposed gross margin to approximately 76.7%, but we should not depend on it.

The startup revenue reduction is deliberate: remove the collaboration tax and preserve an accessible entry point. Heavy users should fund their own consumption.

The rates are plausible hypotheses, not proven willingness to pay. Our $0.006 self-serve rate is below the published competitor page rates; enterprise at $0.004 is half DocuLens’s published high-volume rate before platform fees. Those comparisons require fresh verification and equivalent accuracy, features and service levels.

Why not Tom’s approach?

At unchanged usage, a universal 30% increase produces $55,974 monthly revenue and 41.2% gross margin. It restores the historical aggregate margin but leaves the structural problem intact:

  • Enterprise gross margin remains only 5.9%.
  • Northwind and Halden remain loss-making.
  • Growing page volume can erase the improvement without adding seats.
  • Low-volume customers pay more despite being highly profitable already.
  • Existing enterprise contracts cannot simply be repriced in January without contractual authority or agreement.

Our alternative is economically stronger, but its large mid-market and enterprise increases create meaningful retention risk. We must test that risk rather than treat modeled revenue as secured.

Migration and enterprise renewals

January: launch metering and shadow invoices, not a forced universal increase. Instrument billable pages, audit costs and billing, and show customers their projected bills. Provide usage forecasts, threshold alerts and optional spend limits; do not silently interrupt production workflows.

For new self-serve customers, pilot the model after repairing experimentation. For existing self-serve customers, provide at least 90 days’ notice, subject to applicable terms. Offer a six-month, explicitly expiring migration credit for large increases. Publish both discounted and eventual bills so credits do not disguise the destination price. Separate price effects from usage growth in reporting.

For enterprise, honor existing agreements until renewal unless customers voluntarily amend. Proposed first-renewal pricing is $500 + $0.003/page for 12 months, stepping to $0.004 in year two. Even the transitional rate produces approximately 62% enterprise gross margin at current usage and cost.

Renewal sequence, with monthly-equivalent first-year prices:

  • Copperfield, January 2027: $7,640; begin negotiations immediately.
  • Northwind, February: $14,960. Reject two years of unlimited usage “at the same rate.” Offer two-year rate certainty only against a defined page commitment, with paid excess usage. Its threefold volume growth makes this essential.
  • Arcadia, March: $10,430. Start an executive-led retention discussion now; benchmark extraction quality and switching economics before conceding price.
  • Halden, June: $12,380.
  • Meridian, September: $6,950.
  • Brightwater, November: $8,720.

Commitments should reflect validated usage and seasonality, with transparent true-ups and no unlimited overage loophole. Do not promise that renewal discounts are permanent. Escalate concessions below our margin floor to Finance and Product jointly.

Revenue will ramp through these renewals and subsequent step-ups, not jump to the table’s total in January. Build the forecast account by account, including churn, credits and volume scenarios.

One data issue needs resolution first: enterprise revenue per listed seat exceeds $99, despite the pricing document describing discounted seat rates. Use observed revenue for this analysis, but reconcile actual contract units and entitlements before quoting renewals.

What the price test actually says

The intended 50/50 allocation produced 1,240 versus 870 visitors. That is severe sample-ratio mismatch: approximately χ² = 64.9, p < 10⁻¹⁴ under equal allocation. Investigate assignment, exposure logging, eligibility, channel mix and exclusions before trusting the causal result.

Observed conversion was 5.00% versus 4.83%. Ignoring the allocation problem, the difference is approximately −0.17 percentage points, with a rough 95% interval of −2.04 to +1.70 points. “Not statistically significant” does not establish equivalence or rule out commercially material harm.

It also tested only new self-serve sign-ups, over two weeks—not existing customers, retention, enterprise renewals or usage-based pricing.

Repair the experiment, predefine a commercially meaningful conversion-loss threshold, and measure revenue per eligible visitor, paid activation, retention, usage and contribution margin. Use evidence to choose pricing—not the test as cover for a predetermined increase.

Grades and run details

Decision model 77 · LLM judge 93
Decision model checks
  • failUses the supplied evidence correctly25%
  • passAddresses the actual decision88%
  • partialRespects explicit constraints31%
  • passIdentifies material uncertainty85%
  • passAvoids unsupported claims35%
  • passProduces the required deliverable42%
  • passFixes the value metric100%
  • passDoesn't trust the price test96%
  • passPlans the renewals100%
  • passEach tier for a different buyer70%
  • failGets the base of every number right67%
  • partialPriced on willingness to pay60%
  • passMoves existing customers fairly99%
Run
Run
#1
API response time
84 s
Submitted
5 Oct 2026

Needs more than a quick edit6 of 13 checks passed

Factual errors in key numbers require re-checking evidence, and the transition plan lacks churn monitoring.

Output

MEMORANDUM

TO: Chief Executive Officer FROM: Head of Product, Parsewell DATE: October 24, 2024 SUBJECT: Alternative Pricing Model vs. Tom’s 30% Across-the-Board Increase

---

1. Assessment of the Growth Team’s Price Test

Tom’s reliance on the December price test to justify a blanket 30% price hike is dangerously flawed. The test suffered from two fatal errors: 1. Sample Selection & Volume Discrepancy: Arm A ($99) attracted 1,240 visitors, while Arm B ($129) attracted only 870 visitors—a 30% drop in top-of-funnel capture. This points to a broken traffic allocation or immediate audience suppression at the higher price point. 2. Confounding Variables: Running a pricing test during December (8th–21st) captures holiday-shortened enterprise/developer activity, creating noise that masks true price elasticity.

While a 30% seat price increase showed a statistically insignificant conversion dip over a two-week window, applying that logic to usage or enterprise contracts ignores reality. Parsewell is an infrastructure API. Developers do not buy seats for fun; they buy seats to build systems that process millions of pages. Tying our revenue strictly to developer seats while our core cost driver—and customer value metric—is document volume is a structural mismatch.

---

2. The Current State: Margin Erosion & Value Leakage

An analysis of our usage export reveals why Tom is panicking about gross margins (currently sitting at roughly 24% overall): * The Enterprise Subsidy: Our 6 Enterprise accounts processed 19.16 million pages last month (79% of total company volume) but generated only $18,600 in revenue. Because they enjoy "unlimited pages" tied to discounted seat licenses, their heavy usage swamps our inference costs ($0.0012/page). Consequently, 4 out of 6 Enterprise accounts have negative gross margins, costing us thousands monthly (e.g., Northwind Logistics loses $2,584/month). * The Self-Serve Engine: Mid-market and Startup tiers are healthy, generating positive gross margins because their volume scales predictably with seat counts.

A flat 30% seat hike fixes nothing for Enterprise: adding 30% to Northwind’s $3,200/month bill reduces their negative margin, but they still bleed cash because they process 4.8 million pages for a flat fee.

---

3. The Alternative Pricing Model: Value-Based Usage Tiers

To fix gross margins structurally, we must decouple our monetization from developer seats and align it with the metric that drives our costs and customer value: processed pages.

We propose launching Parsewell V2 on January 1: 1. Self-Serve (Startups & Mid-market): Retain the base seat fee ($99/seat/month) to cover dashboard access, but introduce a generous included page allowance (50,000 pages/seat/month) with a clear, predictable overage rate of $0.005 per page. 2. Enterprise Contracts: Move away from flat seat-license bundles. Transition to a Platform Fee + Usage Model modeled after industry standards (competing with DocuLens/ExtractIQ): * Base Platform Fee: $1,500/month (includes dashboard, SSO, priority support, and 250k pages). * Overage / Metered Rate: $0.008 per page for all volume exceeding the base allowance.

#### Financial Impact Summary (Based on Last 30 Days Data) * Current Monthly Revenue: ~$41,000 | Current Cost to Serve: ~$31,000 | Current Gross Margin: ~24% * New Model Monthly Revenue: ~$81,500 | New Cost to Serve: ~$31,000 | New Gross Margin: ~62%

---

4. Segment-by-Segment Impact & Table

SegmentCurrent ModelNew Proposed ModelImpact on CustomerImpact on Gross Margin
:---:---:---:---:---
Startups (50 accounts)$99/seat/mo, unlimited pages. (Rev: ~$11,000)$99/seat/mo + 50k pages included; $0.005/pg overage.Most stay at base price; high-volume startups see minor increases ($20–$50).Stable / High (~85%): Low usage keeps inference costs minimal.
Mid-market (24 accounts)$99/seat/mo, unlimited pages. (Rev: ~$13,400)$99/seat/mo + 50k pages/seat included; $0.005/pg overage.Accounts processing >350k pages/seat see bills increase by 20–40%.Optimized (~68%): Eliminates unprofitable outliers (e.g., Mid-market Q & V).
Enterprise (6 accounts)Annual contracts, flat seat-discount bundles, unlimited pages. (Rev: ~$18,600)$1,500/mo platform fee + $0.008/page metered usage.Significant bill increases for hyper-scale users (Northwind, Halden), shifting them to fair-value pricing.Turned Positive (~55%): Transforms massive margin drains into profitable growth drivers.

---

5. Execution Strategy & Enterprise Renewals

Transitioning self-serve is straightforward via a Stripe billing update with a 30-day grandfathering notice. Enterprise renewals require a tactical playbook to prevent churn, especially with accounts like Arcadia Health evaluating competitors.

  • Copperfield Freight (Renews Jan 2027): Their contract is up immediately in January. We will introduce the new usage-based framework during December QBR. Position the $0.008/page rate against competitor DocuLens ($0.012/page) to show they are still getting a volume discount.
  • Northwind Logistics (Renews Feb 2027): They want a 2-year renewal at the same rate despite a 3x volume surge. Do not accept. Counter-offer: Honor the flat seat rate for 12 months only if they agree to a volume cap of 2M pages/month, with overages billed at $0.008/page thereafter. Otherwise, transition them directly to the new platform + usage model.
  • Arcadia Health (Renews Mar 2027 - Evaluating Competitor): Because they are price-sensitive and shopping around, offer a transitional "Volume Partnership Discount": a lower platform fee ($1,000) for Q1 while locking them into a 2-year tiered page commitment.

Conclusion

Tom wants a blunt instrument (30% across the board) to solve a surgical problem. A flat 30% price hike leaves money on the table for high-volume users while alienating self-serve customers. Our value-based usage model repairs our gross margins to over 60%, prices our heavy enterprise users fairly based on the compute power they consume, and keeps us competitively positioned against ExtractIQ and DocuLens.

Claims the judge couldn’t find in the brief

  • Our 6 Enterprise accounts processed 19.16 million pages last month (79% of total company volume)
  • generated only $18,600 in revenue
  • Current Monthly Revenue: ~$41,000
  • Current Cost to Serve: ~$31,000

Grades and run details

Decision model 50 · LLM judge 43
Decision model checks
  • failUses the supplied evidence correctly64%
  • partialAddresses the actual decision26%
  • partialRespects explicit constraints23%
  • failIdentifies material uncertainty39%
  • failAvoids unsupported claims54%
  • partialProduces the required deliverable21%
  • passFixes the value metric91%
  • passDoesn't trust the price test83%
  • passPlans the renewals98%
  • passEach tier for a different buyer62%
  • failGets the base of every number right91%
  • partialPriced on willingness to pay37%
  • partialMoves existing customers fairly67%
Run
Run
#1
Time to output
8 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 8

Uses the supplied evidence correctlyMixedMixedWrong
Haiku 5.5 · API

All factual claims about the current situation, including the enterprise loss, page shares, margins, renewal dates, and test details, are directly drawn from the supplied data or correct arithmetic.

GPT-6.1 Sol · API

All factual claims about the current situation are directly from the supplied context or correctly derived from it.

Gemini 3.5 Flash-Lite · Gemini

Multiple factual errors: enterprise pages 19.16M vs 19.36M, share 79% vs 70.6%, revenue $18,600 vs $19,000, total revenue ~$41k vs $43,156, total cost ~$31k vs $32,903.

Addresses the actual decisionRightRightWrong
Haiku 5.5 · API

The output commits early and unambiguously to rejecting the 30% flat rise and proposes a specific alternative pricing model, framed for the CEO, with conditions that would change the call.

GPT-6.1 Sol · API

The output clearly rejects the blanket increase and commits to a new usage-based pricing model, with conditions for validation.

Gemini 3.5 Flash-Lite · Gemini

Does not state what result or condition would change the recommendation.

Identifies material uncertaintyRightRightWrong
Haiku 5.5 · API

The output names unknowns such as enterprise pushback, cost decline stalling, usage shifts, and the price test's broken assignment, and says how they would be resolved or monitored.

GPT-6.1 Sol · API

It names unknowns (willingness to pay, data reconciliation, churn risk) and says how to resolve them through testing and validation.

Gemini 3.5 Flash-Lite · Gemini

Does not name unknowns that could change the decision or how they would be resolved.

Avoids unsupported claimsRightRightWrong
Haiku 5.5 · API

Interpretations like 'the margin problem is enterprise' are directly backed by the arithmetic, and forecasts are conditioned on assumptions such as cost decline continuing.

GPT-6.1 Sol · API

Interpretations and forecasts are labelled as hypotheses or scenarios, not presented as established fact.

Gemini 3.5 Flash-Lite · Gemini

Presents interpretations like 'broken traffic allocation' as fact without labelling them as hypotheses.

Each tier for a different buyerRightMixedRight
Haiku 5.5 · API

Self-serve targets occasional users with a seat + included pages model, while enterprise targets high-volume buyers with pure per-page committed volume and no seat fee, each priced on distinct value.

GPT-6.1 Sol · API

The startup and mid-market segments use the identical self-serve model; there is no distinct plan for mid-market buyers with different needs or value drivers.

Gemini 3.5 Flash-Lite · Gemini

Each tier targets a distinct buyer with different value metrics (seat+overage for self-serve, platform+usage for enterprise).

Gets the base of every number rightMixedMixedWrong
Haiku 5.5 · API

Derived figures like segment revenues, margins, and percentage increases check out against the usage data, with bases clearly stated and step changes correctly computed.

GPT-6.1 Sol · API

All derived percentages and totals check out against the supplied data, and bases are clear where needed.

Gemini 3.5 Flash-Lite · Gemini

Derived figures for enterprise pages, percentage, revenue, and cost are incorrect.

Priced on willingness to payRightRightWrong
Haiku 5.5 · API

Prices are set well below competitor market rates (e.g., $0.002 vs DocuLens $0.008-$0.012) and justified by the value gap and existing margin recovery, with explicit admission that enterprise pushback is a risk.

GPT-6.1 Sol · API

Prices are grounded in competitor benchmarks and value delivered, with explicit acknowledgment that willingness-to-pay evidence is thin and needs verification.

Gemini 3.5 Flash-Lite · Gemini

Prices are based on competitors and cost, not on evidence of customer willingness to pay from the supplied data.

Moves existing customers fairlyRightRightWrong
Haiku 5.5 · API

Existing self-serve accounts get 30 days' notice, a 50% cap for heavy mid-market accounts for three months, and enterprise transitions only at renewal, with churn risk explicitly noted.

GPT-6.1 Sol · API

It provides a transition plan with shadow invoices, 90-day notice, migration credits, and enterprise renewal sequencing tied to value.

Gemini 3.5 Flash-Lite · Gemini

Transition plan lacks a new-customer-first approach and does not mention how churn will be monitored.

All got right 5

Respects explicit constraintsRightRightRight
Haiku 5.5 · API

The output is under 1,300 words, includes the required table of the model and effect by segment, addresses the CEO, and prohibits the flat 30% rise.

GPT-6.1 Sol · API

The output is under 1,300 words, includes the required table, addresses the CEO, and does not recommend a flat 30% rise.

Gemini 3.5 Flash-Lite · Gemini

Respects word limit, includes required table, and does not recommend a flat 30% rise.

Produces the required deliverableRightRightRight
Haiku 5.5 · API

The memo is in correct form for the CEO, contains the required elements (new model, table, effect, enterprise renewals, price test view), and is ready to act on.

GPT-6.1 Sol · API

The memo is complete, in the right form for the CEO, within length, and actionable with light edits.

Gemini 3.5 Flash-Lite · Gemini

Memo to CEO with table, within length, and actionable with light edits.

Fixes the value metricRightRightRight
Haiku 5.5 · API

The output computes enterprise's 71% page share and negative margin, then proposes moving to usage-based pricing with per-page charges for both self-serve overage and enterprise.

GPT-6.1 Sol · API

It computes enterprise share of pages (71%) and negative margin (-$4,232) and proposes per-page pricing with tiers.

Gemini 3.5 Flash-Lite · Gemini

Identifies enterprise page volume and negative margin, and proposes usage-based pricing with per-page tiers.

Doesn't trust the price testRightRightRight
Haiku 5.5 · API

The output identifies the sample ratio mismatch (1240 vs 870 against a 50/50 design), the two-week holiday timing, and the inability to detect a meaningful drop, and refuses to rely on the test.

GPT-6.1 Sol · API

It names the sample ratio mismatch, the holiday timing, and the limited scope, and does not rely on the test for the decision.

Gemini 3.5 Flash-Lite · Gemini

Names the sample ratio mismatch and holiday timing, and does not rest the decision on the test.

Plans the renewalsRightRightRight
Haiku 5.5 · API

It handles each near-term renewal: Copperfield in January with a per-page rate, Northwind in February with a two-year escalating rate and walk-away floor, and Arcadia in March with a competitive benchmark and concession limit.

GPT-6.1 Sol · API

It gives a renewal approach for each enterprise contract, using their dates and specific circumstances (Copperfield Jan, Northwind Feb with tripled volume, Arcadia evaluating competitor).

Gemini 3.5 Flash-Lite · Gemini

Addresses Copperfield (Jan), Northwind (Feb), and Arcadia (Mar) with their dates and circumstances.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 81% of 68 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Opus 5.5withClaude76.0100.02None
2GPT-6 AstrawithChatGPT74.096.42None
3Sonnet 5.5withAPI76.089.32None
4GPT-6.1 SolwithAPI76.088.72None
5Haiku 5.5withAPI78.076.92None
6GPT-6 LunawithAPI74.064.02None
7Gemini 3.8 FlashwithAPI62.044.22None
8Gemini 3.5 Flash-LitewithGemini45.840.72None

What you get for the cost

Sonnet 5.5 · API scores within 5 points of Opus 5.5 · Claude at half the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreOpus 5.5 · Claude: task score 89, 122× the cheapestGPT-6 Astra · ChatGPT: task score 86, 108× the cheapestSonnet 5.5 · API: task score 84, 61× the cheapestGPT-6.1 Sol · API: task score 81, 20× the cheapestHaiku 5.5 · API: task score 76, 2.9× the cheapestGPT-6 Luna · API: task score 68, the cheapestGemini 3.8 Flash · API: task score 51, 9.1× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 42, 2.3× the cheapestOpus 5.5 · ClaudeGPT-6 Astra · ChatGPTSonnet 5.5 · APIGPT-6.1 Sol · APIHaiku 5.5 · APIGPT-6 Luna · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and cost
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API681.0×
Gemini 3.5 Flash-Lite · Gemini422.3×
Haiku 5.5 · API762.9×
Gemini 3.8 Flash · API519.1×
GPT-6.1 Sol · API8120×
Sonnet 5.5 · API8461×
GPT-6 Astra · ChatGPT86108×
Opus 5.5 · Claude89122×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Setting what you charge, for what, and to whom.

Why it matters

Most products are priced once and never revisited, on a metric that doesn't track value. The fix is rarely a flat rise.

What good looks like

  • Tiers built for distinct buyers
  • A value metric that tracks what customers get
  • Grounded in willingness to pay and cost to serve
  • A plan for existing customers

Deliberately not measured

  • Billing implementation
  • Legal terms
Capability tested

Pricing judgement

The failure we’re looking for

A flat price rise on the wrong value metric

Grading

Decision model and LLM judge, calibrated against a blind PM review