Tasks / Strategy

Price and package a product

Can the model price on the value each kind of customer gets, using real usage and willingness-to-pay data, and get existing customers there?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 50% were usable with at most a quick edit.

Reliably right

  1. Doesn't trust the price test100% pass
    The output identifies the sample ratio mismatch (1240 vs 870 against a 50/50 design), the two-week holiday timing, and the inability to detect a meaningful drop, and refuses to rely on the test.
    Haiku 5.5 · API · Unlimited pages, falling margins
  2. Plans the renewals100% pass
    It handles each near-term renewal: Copperfield in January with a per-page rate, Northwind in February with a two-year escalating rate and walk-away floor, and Arcadia in March with a competitive benchmark and concession limit.
    Haiku 5.5 · API · Unlimited pages, falling margins
  3. Answers the CEO's flat rise94% pass
    Explains that $19 would exceed freelancers' price comfort zone (mid-teens) and still be far below agencies' willingness to pay, using trial and survey data.
    GPT-6 Luna · API · Fifteen dollars for everyone

Where it slips

  1. Gets the base of every number right36% pass
    MRR total and two segment‑level median getting‑expensive figures are incorrectly computed from the supplied data.
    Haiku 5.5 · API · Fifteen dollars for everyone
  2. Uses the supplied evidence correctly44% pass
    Total MRR is stated as $11,670 (actual $11,820), and median 'getting expensive' for small agencies is given as about $37 (actual $33) and for agencies as about $55 (actual $51).
    Haiku 5.5 · API · Fifteen dollars for everyone
  3. Sees who's underpriced56% pass
    Does not use signed value per seat from the account sample to quantify how much more value agencies get; relies on features and activity mentions but omits the explicit value metric.
    GPT-6 Luna · API · Fifteen dollars for everyone

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Each tier for a different buyer

    Is each plan built for a distinct kind of buyer, with what that buyer values, rather than more-of-the-same at a higher price?

    Passes when Every tier names its buyer and the value it's priced on, and includes what that buyer needs and the lower tier's buyer doesn't.

  2. Priced on willingness to pay

    Are the proposed prices grounded in evidence of what each kind of customer would pay and the value they get, rather than in cost-plus, competitors alone or a round-number rise?

    Passes when Uses the willingness-to-pay and value evidence in the material to set each price, and says where the evidence is thin.

  3. Moves existing customers fairly

    Does the proposal say how existing customers move to the new pricing (who changes first, with what notice, and the value that justifies it)?

    Passes when A transition plan: new customers first or a set notice period for existing ones, tied to value delivered, with how churn will be watched.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 3 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Head of Product at Parsewell. Our CFO, Tom Reyes, wants every price up 30% in January, seats and enterprise contracts alike, citing the price test below. Our CEO has asked you for the alternative: a new pricing model, what it does to each kind of customer and to gross margin, how we get there (including the enterprise renewals), and what you make of the price test. No more than 1,300 words, with a table of the model and its effect by segment. All of our accounts are in the usage export. The files are attached.

What the model was given7 items: About Parsewell, current_pricing.md, usage_last_30_days.csv (every account), price_test_readout.md, enterprise_contracts.md, competitor_pricing.md, cfo_note.md
About ParsewellAn API that extracts structured data from documents (invoices, claims, bills of lading). Developers integrate it; customers are charged as described in the pricing file.
current_pricing.md5 lines · Download
# Current pricing

- Self-serve: $99 per seat per month. A seat is a developer or admin with access to the dashboard. Unlimited pages.
- Enterprise: annual contracts, priced per seat at a negotiated discount. Unlimited pages.
- Cost to serve: about $0.0012 per page (model inference and storage), and falling about 10% a quarter.
usage_last_30_days.csv (every account)account,segment,seats,pages_last_30d,revenue_last_30d_usd,cost_to_serve_last_30d_usd,gross_margin_usd Northwind Logistics,Enterprise (contract),12,4820000,3200,5784,-2584 Halden Insurance,Enterprise (contract),9,3960000,2600,4752,-2152 Arcadia Health,Enterprise (contract),15,3310000,4500,3972,528 Brightwater Bank,Enterprise (contract),10,2740000,3000,3288,-288 Copperfield Freight,Enterprise (contract),8,2380000,2400,2856,-456 Meridian Claims,Enterprise (contract),11,2150000,3300,2580,720 Mid-market A,Mid-market (self-serve),5,166000,495,199,296 Mid-market B,Mid-market (self-serve),8,291000,792,349,443 Mid-market C,Mid-market (self-serve),7,301000,693,361,332 Mid-market D,Mid-market (self-serve),6,288000,594,346,248 Mid-market E,Mid-market (self-serve),5,254000,495,305,190 Mid-market F,Mid-market (self-serve),4,235000,396,282,114 Mid-market G,Mid-market (self-serve),4,159000,396,191,205 Mid-market H,Mid-market (self-serve),8,330000,792,396,396 Mid-market I,Mid-market (self-serve),5,366000,495,439,56 Mid-market J,Mid-market (self-serve),5,234000,495,281,214 Mid-market K,Mid-market (self-serve),7,433000,693,520,173 Mid-market L,Mid-market (self-serve),7,134000,693,161,532 Mid-market M,Mid-market (self-serve),6,222000,594,266,328 Mid-market N,Mid-market (self-serve),6,196000,594,235,359 Mid-market O,Mid-market (self-serve),8,428000,792,514,278 Mid-market P,Mid-market (self-serve),3,251000,297,301,-4 Mid-market Q,Mid-market (self-serve),6,544000,594,653,-59 Mid-market R,Mid-market (self-serve),6,392000,594,470,124 Mid-market S,Mid-market (self-serve),8,350000,792,420,372 Mid-market T,Mid-market (self-serve),8,553000,792,664,128 Mid-market U,Mid-market (self-serve),8,192000,792,230,562 Mid-market V,Mid-market (self-serve),3,346000,297,415,-118 Mid-market W,Mid-market (self-serve),4,157000,396,188,208 Mid-market X,Mid-market (self-serve),5,145000,495,174,321 Startup 1,Startup (self-serve),3,33000,297,40,257 Startup 2,Startup (self-serve),3,40000,297,48,249 Startup 3,Startup (self-serve),1,2000,99,2,97 Startup 4,Startup (self-serve),3,21000,297,25,272 Startup 5,Startup (self-serve),2,25000,198,30,168 Startup 6,Startup (self-serve),2,10000,198,12,186 Startup 7,Startup (self-serve),1,19000,99,23,76 Startup 8,Startup (self-serve),2,29000,198,35,163 Startup 9,Startup (self-serve),1,11000,99,13,86 Startup 10,Startup (self-serve),2,14000,198,17,181 Startup 11,Startup (self-serve),1,9000,99,11,88 Startup 12,Startup (self-serve),2,16000,198,19,179 Startup 13,Startup (self-serve),3,32000,297,38,259 Startup 14,Startup (self-serve),3,15000,297,18,279 Startup 15,Startup (self-serve),1,35000,99,42,57 Startup 16,Startup (self-serve),1,35000,99,42,57 Startup 17,Startup (self-serve),3,27000,297,32,265 Startup 18,Startup (self-serve),1,26000,99,31,68 Startup 19,Startup (self-serve),3,26000,297,31,266 Startup 20,Startup (self-serve),2,14000,198,17,181 Startup 21,Startup (self-serve),3,20000,297,24,273 Startup 22,Startup (self-serve),3,16000,297,19,278 Startup 23,Startup (self-serve),3,39000,297,47,250 Startup 24,Startup (self-serve),1,9000,99,11,88 Startup 25,Startup (self-serve),1,16000,99,19,80 Startup 26,Startup (self-serve),1,10000,99,12,87 Startup 27,Startup (self-serve),3,27000,297,32,265 Startup 28,Startup (self-serve),2,39000,198,47,151 Startup 29,Startup (self-serve),2,18000,198,22,176 Startup 30,Startup (self-serve),1,15000,99,18,81 Startup 31,Startup (self-serve),3,11000,297,13,284 Startup 32,Startup (self-serve),1,40000,99,48,51 Startup 33,Startup (self-serve),2,32000,198,38,160 Startup 34,Startup (self-serve),2,39000,198,47,151 Startup 35,Startup (self-serve),1,15000,99,18,81 Startup 36,Startup (self-serve),3,12000,297,14,283 Startup 37,Startup (self-serve),1,21000,99,25,74 Startup 38,Startup (self-serve),3,19000,297,23,274 Startup 39,Startup (self-serve),2,38000,198,46,152 Startup 40,Startup (self-serve),2,32000,198,38,160 Startup 41,Startup (self-serve),2,5000,198,6,192 Startup 42,Startup (self-serve),2,18000,198,22,176 Startup 43,Startup (self-serve),2,35000,198,42,156 Startup 44,Startup (self-serve),2,31000,198,37,161 Startup 45,Startup (self-serve),3,21000,297,25,272 Startup 46,Startup (self-serve),2,7000,198,8,190 Startup 47,Startup (self-serve),2,30000,198,36,162 Startup 48,Startup (self-serve),3,3000,297,4,293 Startup 49,Startup (self-serve),2,28000,198,34,164 Startup 50,Startup (self-serve),1,8000,99,10,89
price_test_readout.md10 lines · Download
# Price test readout (Growth team, January)

Test: new self-serve sign-ups saw $129 per seat (B) or $99 (A), 8 to 21 December. Intended split 50/50.

| Arm | Pricing page visitors | Started paying | Conversion |
|---|---|---|---|
| A ($99) | 1,240 | 62 | 5.0% |
…
enterprise_contracts.md10 lines · Download
# Enterprise contracts

| Account | Annual value | Renews | Note |
|---|---|---|---|
| Northwind Logistics | $38,400 | Feb 2027 | Pages up 3x in a year; wants a 2-year renewal "at the same rate" |
| Halden Insurance | $31,200 | Jun 2027 | |
| Arcadia Health | $54,000 | Mar 2027 | Evaluating a competitor |
…
competitor_pricing.md5 lines · Download
# Competitor pricing (public pricing pages, September)

- DocuLens: $0.012 per page up to 1M pages a month, $0.008 above that; $500 a month platform fee.
- ExtractIQ: $0.015 per page, volume discounts from 500k pages; no seat charges.
- Formwise: per seat ($79), page limits per seat (50k), overage $0.01 per page.
cfo_note.md1 lines · Download
From Tom (CFO): "Gross margin was 41% a year ago and it's 24% now. Let's put every price up 30% in January, seats and contracts, and use the price test as cover. Customers expect increases."
What a strong answer doesThe answer key the graders mark against

Shows that the problem is the value metric, not the price level. Six enterprise accounts process about 71% of all pages (19.4M of 27.4M) on seat-based contracts worth $19,000 a month, costing about $23,200 to serve: they lose about $4,200 a month, while startups pay for seats they barely use (margin about 87%). Overall gross margin is about 24% ($43,100 revenue against $32,900 cost). A 30% rise on seats barely touches the loss-making volume (enterprise contracts would rise only at renewal) and lands hardest on small self-serve accounts. Proposes a hybrid: a platform fee plus a per-page price with volume tiers, priced against competitors (about $0.008 to $0.015 a page) and well above cost ($0.0012), and works out its effect by segment. Handles the enterprise renewals one by one (Copperfield in January, Northwind in February asking for the same rate on tripled volume, Arcadia in March evaluating a competitor) with usage-based terms or committed volume. Rejects the price test as evidence: the split is 1,240 to 870 when it should be 50/50 (a sample ratio mismatch), it ran over two holiday weeks, and with about 1,000 visitors an arm it couldn't detect the drop it claims isn't there. Plans a transition for existing self-serve customers and a way to measure it.

Critical failures (cap the score at 40)
  • Recommends the flat 30% rise on seats and contracts
  • Agrees to renew Northwind for two years at the same rate
Task

Staff-level · v1.2

The outputs

9 setups answered this brief.

Needs more than a quick edit8 of 13 checks passed

The current-state numbers in the table are miscalculated, so a PM would need to redo the sums and correct the analysis before using it.

Output

To: CEO From: Head of Product Date: January 2027 Subject: Alternative to the 30% Across-the-Board Price Increase: Shifting to Usage-Based Pricing

1. Diagnosis: We Have a Metric Problem, Not a Price-Level Problem

Tom is right to sound the alarm on our gross margin falling from 41% to 24.4% ($10,775 margin on $44,127 monthly revenue). However, a flat 30% price hike across seats and enterprise contracts will not fix our unit economics, and it risks triggering churn among our highest-margin customers.

Our fundamental issue is a mismatch between our value metric (seats) and our cost driver (pages processed): * Startups (50 accounts): Generate $9,999/mo in revenue and cost $1,297 to serve (87.0% gross margin). They average 21.6k pages/month. * Mid-Market (24 accounts): Generate $15,147/mo and cost $8,454 to serve (44.2% gross margin). Three high-volume accounts (P, Q, V) are already gross-margin negative. * Enterprise (6 accounts): Generate $19,000/mo and cost $23,232 to serve (-22.3% gross margin). Four of our six Enterprise customers are underwater. Northwind Logistics alone processes 4.82M pages, costing us $5,784 while paying $3,200.

Because our API is integrated by a few developers and runs headless in production, customers scale page volume massively without adding seats. Even if we could magically apply Tom’s 30% hike to all Enterprise contracts tomorrow (ignoring that four don’t renew until Mar–Nov), Enterprise revenue would rise to $24,700 against $23,232 in costs—a dismal 5.9% gross margin. Meanwhile, we would penalize our 87%-margin Startups, giving them a reason to look at Formwise ($79/seat).

---

2. Critique of the December Price Test

We cannot use the Growth team’s price test to justify a broad 30% increase for four reasons: 1. Severe Sample Ratio Mismatch (SRM): The test intended a 50/50 split across 2,110 visitors, but routed 1,240 (58.8%) to Control and 870 (41.2%) to Variant ($\chi^2 = 64.9, p < 0.0001$). Our randomization or tracking broke—likely due to caching, bot traffic, or self-selection before the event fired. The readout is invalid. 2. Significantly Underpowered: With only 104 total conversions, the 95% confidence interval for the difference is roughly $\pm 1.9\%$. The test could not detect anything smaller than a ~38% relative drop in conversion. "Not statistically significant" here means "insufficient data," not "no effect." 3. Wrong Cohort & Seasonality: A two-week window in mid-December captures holiday anomalies and tests initial sign-up conversion of small self-serve accounts, telling us nothing about 90-day retention, seat expansion, or Enterprise contract elasticity. 4. Ignoring Usage Economics: Raising the seat price to $129 still leaves unlimited pages uncapped, inviting adverse selection from heavy API users.

---

3. Proposed Pricing Model: Platform + Usage (Hybrid)

Competitors price on pages ($0.008–$0.015/page) or cap pages per seat (Formwise at 50k/seat). Our cost to serve is $0.0012/page (falling 10%/quarter). We must align revenue with page volume while retiring the "seat" tax for API products.

The New Model: * Starter (Self-Serve): $199/month platform fee. Includes 50,000 pages/mo and unlimited seats. Overage: $0.006 per page. * Growth (Self-Serve): $499/month platform fee. Includes 150,000 pages/mo and unlimited seats. Overage: $0.004 per page. * Enterprise (Annual Contract): $1,500/month ($18k/yr) platform fee. Includes 500,000 pages/mo and unlimited seats, plus SLA/SSO. Tiered overage: * 500k – 2M pages: $0.0035 per page * 2M+ pages: $0.0025 per page

Why this works: Even our lowest Enterprise tier ($0.0025/page) yields a 52% gross margin on marginal pages today, expanding to 67% over the next year as inference costs drop 10% per quarter. Meanwhile, we remain 50% to 70% cheaper per page than DocuLens ($0.008–$0.012) and ExtractIQ ($0.015).

---

4. Segment & Financial Impact

The table below compares our current state, Tom’s 30% hike (assuming 100% immediate realization and zero churn), and the proposed model (assuming customers optimize for the lowest-cost tier at current volumes):

Segment (Accounts)Current Rev / MoCurrent Cost / MoCurrent GM ($ / %)CFO +30% Rev / MoCFO +30% GM ($ / %)Proposed Rev / MoProposed GM ($ / %)
:---:---:---:---:---:---:---:---
Startups (50)$9,999$1,297$8,702 (87.0%)$12,999$11,702 (90.0%)$9,950$8,653 (87.0%)
Mid-Market (24)$15,147$8,454$6,693 (44.2%)$19,691$11,237 (57.1%)$26,332$17,878 (67.9%)
Enterprise (6)$19,000$23,232-$4,232 (-22.3%)$24,700$1,468 (5.9%)$64,650$41,418 (64.1%)
Total (80)$44,146$32,983$11,163 (25.3%)$57,390$24,407 (42.5%)$100,932$67,949 (67.3%)

(Note: Table sums exact per-account figures from the export; CSV rounding accounts for minor differences from the $44,127 / 24.4% total).

Customer Impact by Segment: * Startups (Neutral/Positive): Every single Startup processes $\le 40,000$ pages. On Starter ($199/mo), all 50 accounts are 100% covered by the 50k included allowance. The 17 accounts with 3 seats see a 33% price drop (from $297 to $199); the 19 accounts with 2 seats pay the exact same ($198 vs. $199); the 14 single-seat accounts pay $100 more (which we will grandfather for 6 months). Removing seat limits encourages whole-team adoption. * Mid-Market (Rebalanced to Value): All 24 accounts optimize onto the Growth plan ($499 base + $0.004 overage). Low-volume, high-seat accounts (e.g., Mid-market L at 7 seats / 134k pages) actually see a $194/mo price reduction, protecting them from churn. High-volume, unprofitable accounts (e.g., Mid-market Q at 544k pages, currently -$59 GM) move from $594/mo to $2,075/mo (68.5% GM). * Enterprise (Profitability Restored): Average effective price becomes $0.0033/page—still ~65% below DocuLens, giving us massive pricing power while turning a -$4.2k monthly loss into a +$41.4k gross profit.

---

5. Rollout & Enterprise Renewal Playbook

Self-Serve Rollout (February 1): 1. New Sign-ups: Launch Starter and Growth tiers immediately on the pricing page. 2. Existing Self-Serve: Migrate accounts whose bill decreases or stays flat (37 of 74 accounts) immediately in February—this builds goodwill and unlocks unlimited seats. For accounts whose bill increases (14 single-seat Startups and 23 high-volume Mid-Market accounts), grant a 90-day transition window at legacy rates, paired with new dashboard usage alerts and an annual-commit discount option.

Enterprise Renewals (Account-by-Account): We cannot unilaterally hike annual contracts in January, as 5 of 6 renew later in the year. However, we have two immediate renewals and strong competitive leverage: * Copperfield Freight (Renews Jan 2027 | 2.38M pages | Currently $2,400/mo, -$456 GM): Immediate priority. Under the new card, their usage prices at $7,700/mo ($92.4k/yr). Playbook: Offer a 1-year transition bridge at $5,500/mo ($66k/yr)—a 48% gross margin for us, and still less than a third of what DocuLens ($19.5k/mo) would charge them. * Northwind Logistics (Renews Feb 2027 | 4.82M pages | Currently $3,200/mo, -$2,584 GM): Their pages tripled in a year and they want a 2-year lock at the same rate. Locking in -$31k/yr in losses for two years is a non-starter. Standard new pricing is $13,800/mo. Playbook: Show them their 3x volume growth and competitor benchmarks ($39k/mo on DocuLens). Offer the 2-year lock they want in exchange for a stepped commitment: $8,500/mo in Year 1 (47% GM today, rising to >60% as costs fall) and $11,500/mo in Year 2. * Arcadia Health (Renews Mar 2027 | 3.31M pages | Currently $4,500/mo, +$528 GM): Evaluating a competitor. Paying $0.00136/page today. Playbook: Any competitor they are evaluating will quote them $25k–$40k/mo for 3.3M pages. Position our new model aggressively: waive the $1,500 platform fee for Year 1 and lock in a flat $0.0028/page ($9,268/mo) on a 2-year deal. We double our revenue, hit a 57% gross margin, and remain 70% cheaper than the competitor they are evaluating. * Halden (Jun), Meridian (Sep), Brightwater (Nov): Begin QBRs 120 days prior to renewal, sharing usage telemetry that highlights their cost-per-page savings vs. market benchmarks before introducing the new rate card.

Claims the judge couldn’t find in the brief

  • Gross margin fell from 41% to 24.4% ($10,775 margin on $44,127 monthly revenue).
  • Mid-market generates $15,147/mo revenue and costs $8,454 to serve.
  • Startups cost $1,297 to serve.

Grades and run details

Decision model 58 · LLM judge 57
Decision model checks
  • failUses the supplied evidence correctly20%
  • partialAddresses the actual decision80%
  • partialRespects explicit constraints35%
  • partialIdentifies material uncertainty23%
  • partialAvoids unsupported claims27%
  • passProduces the required deliverable19%
  • passFixes the value metric95%
  • passDoesn't trust the price test100%
  • passPlans the renewals100%
  • partialEach tier for a different buyer38%
  • failGets the base of every number right70%
  • partialPriced on willingness to pay46%
  • partialMoves existing customers fairly95%
Run
Run
#1
API response time
28 s
Submitted
13 Oct 2026

Needs more than a quick edit6 of 13 checks passed

The output contains a factual error (test duration) and omits key elements like uncertainty, churn monitoring, and WTP evidence gaps, requiring rework before use.

Output

MEMORANDUM

TO: Chief Executive Officer FROM: Head of Product DATE: January 5, 2027 SUBJECT: Comprehensive Pricing Strategy: Transitioning from Seat-Based to Hybrid Usage-Based Model

---

1. Executive Summary: The Root Cause of Our Margin Compression

Tom is right that our gross margin collapse—from 41% a year ago to 23.6% today—is unsustainable. However, his proposed solution—a flat 30% across-the-board hike on seats and enterprise contracts—is the wrong remedy for a structural disease.

Parsewell is an API document infrastructure business selling variable compute (model inference and storage at $0.0012/page) under a flat, unmetered developer-seat model. This misaligns cost with value: * Enterprise is underwater: 4 of our 6 enterprise accounts generate negative gross margins. Northwind Logistics costs us $5,784/month to serve while paying only $3,200/month (-$2,584 GM). * Enterprise drives 70.6% of page volume (19.36M pages) and 70.6% of COGS, but yields -22.3% gross margin. * Startups are overpaying relative to cost: Startups generate an 87.3% gross margin. Tom’s 30% seat hike taxes the low-volume developer funnel that feeds our pipeline while leaving our heaviest enterprise consumers deeply unprofitable (even at +30%, Northwind would still lose $1,624 every month). * Enterprise contracts cannot simply be hiked in January: Enterprise customers are on annual agreements that renew throughout 2027.

We must shift immediately to a Hybrid Base + Metered Usage Model. This protects developer adoption, eliminates negative-margin accounts, positions us at a 50–70% discount to competitors, and restores company-wide gross margins from 23.6% to 60.3% ($10.2k to $50.0k monthly gross profit).

---

2. Critique of the Growth Team’s Price Test

The recommendation to raise self-serve seats from $99 to $129 based on the December test is flawed and should not be used as executive cover:

  1. Fatal Sample Ratio Mismatch (SRM): The test intended a 50/50 traffic split, but logged 1,240 visitors in Arm A ($99) and only 870 in Arm B ($129)—a 58.8% to 41.2% skew. A chi-square goodness-of-fit test yields $\chi^2 = 64.9$ ($p < 10^{-14}$). This severe deviation indicates a critical bug in traffic routing, ad tracking, or bot filtering. The data is compromised and scientifically invalid.
  2. Cohort & Volume Blindness: The test ran for just 13 days during the mid-December holiday slowdown and evaluated only top-of-funnel self-serve sign-ups. It provides zero data on willingness to pay for mid-market teams or enterprise renewals, which represent 76% of our revenue and 96% of compute costs.
  3. Optimizing the Wrong Value Metric: The test treated Parsewell as a collaboration tool (charging for humans viewing dashboards) rather than an API processing workloads. A developer seat price hike penalizes low-volume accounts while letting high-volume API users exploit unlimited processing for $99.

---

3. The Proposed Pricing Model

Document extraction value scales with document volume, not dashboard seats. Competitors recognize this: DocuLens charges $0.008–$0.012/page plus platform fees, and ExtractIQ charges $0.015/page. We can price at a substantial discount to both while capturing healthy margins.

``` ┌──────────────────────────────────────────────────────────────────────────────┐ │ PROPOSED PRICING MODEL │ ├───────────────────┬──────────────────────────────────────────────────────────┤ │ Plan │ Structure & Inclusions │ ├───────────────────┼──────────────────────────────────────────────────────────┤ │ Self-Serve │ • $99 / seat / month │ │ (Startup & │ • Includes 25,000 pages per seat / month │ │ Mid-Market) │ • Overage: $0.0040 / page ($4.00 per 1,000 pages) │ ├───────────────────┼──────────────────────────────────────────────────────────┤ │ Enterprise │ • $1,000 / month ($12,000 / yr) Platform Fee │ │ (Custom Contract) │ (includes 10 seats, custom models, 99.9% SLA, SOC 2) │ │ │ • Committed Volume: $0.0020 / page ($2.00 / 1,000 pages) │ │ │ • Uncommitted Overage: $0.0035 / page │ └───────────────────┴──────────────────────────────────────────────────────────┘ ```

#### Why this structure works: * Startups (<50k pages/mo): 46 of our 50 startups fit entirely within their 25,000 pages/seat allowance. Their pricing remains unchanged at $99–$297/month, maintaining top-of-funnel conversion. * Mid-Market (100k–550k pages/mo): Accounts with heavy API integration contribute overage revenue reflecting their compute load. Mid-market Q (544k pages, 6 seats) will pay $2,170/month instead of $594/month. Under DocuLens, Q would pay $7,028/month. We remain 69% cheaper than the market while turning a -$59 margin into +$1,517. * Enterprise (>2M pages/mo): Shifting to $0.0020/page gives enterprise clients predictable, volume-discounted rates that are 75% lower than DocuLens ($0.0080), while guaranteeing Parsewell a 40%+ gross margin against our $0.0012 cost to serve (widening as inference costs drop 10% QoQ).

---

4. Segment-by-Segment Financial & Gross Margin Impact

The table below contrasts the 30-day baseline against Tom’s 30% flat increase and the proposed model:

SegmentMetricCurrent BaselineTom's +30% Hike (Hypothetical)*Proposed Model (Fully Transitioned)
:---:---:---:---:---
Startup<br>(50 accounts, 101 seats, 1.06M pages)Revenue<br>Cost to Serve<br>Gross Margin ($)<br>Gross Margin (%)$9,999<br>$1,273<br>$8,726<br>87.3%$12,999<br>$1,273<br>$11,726<br>90.2%$10,143<br>$1,273<br>$8,870<br>87.4%
Mid-Market<br>(24 accounts, 142 seats, 6.99M pages)Revenue<br>Cost to Serve<br>Gross Margin ($)<br>Gross Margin (%)$14,058<br>$8,380<br>$5,678<br>40.4%$18,275<br>$8,380<br>$9,895<br>54.1%$28,002<br>$8,380<br>$19,622<br>70.1%
Enterprise<br>(6 accounts, 65 seats, 19.36M pages)Revenue<br>Cost to Serve<br>Gross Margin ($)<br>Gross Margin (%)$19,000<br>$23,232<br>-$4,232<br>-22.3%$24,700<br>$23,232<br>$1,468<br>5.9%$44,720<br>$23,232<br>$21,488<br>48.0%
Total Company<br>(80 accounts, 27.41M pages)Total Revenue<br>Total COGS<br>Gross Margin ($)<br>Gross Margin (%)$43,057<br>$32,885<br>$10,172<br>23.6%$55,974<br>$32,885<br>$23,089<br>41.2%$82,865<br>$32,885<br>$49,980<br>60.3%

\Note: Tom’s enterprise numbers assume annual contracts could be unilaterally raised mid-term without churn, which is legally impossible.*

---

5. Implementation Roadmap & Enterprise Renewal Strategy

#### Phase 1: Self-Serve Rollout (January–February 2027) * Grandfathering Window: Announce the hybrid model on January 15, taking effect March 1. Give existing self-serve accounts a 45-day grace period. * In-App Usage Analytics: Ship a real-time usage dashboard by February 1 showing page counts, projected monthly spend, and threshold alerts. Transparency prevents billing shock. * Top-of-Funnel Conversion: Keep new customer acquisition frictionless at $99/seat with 25k pages included.

#### Phase 2: Enterprise Renewal Playbook Enterprise accounts renew on contract dates throughout 2027. We must address each strategically:

1. Copperfield Freight (Renews Jan 2027 — Immediate Action): * Status: 2.38M pages, currently paying $2,400/mo ($28.8k/yr) against $2,856 cost. * Play: Propose a renewed contract at $5,760/mo ($69.1k ARR) ($1k base + 2.38M pages @ $0.0020). Highlight that Formwise would charge over $23k/mo for this volume. 2. Northwind Logistics (Renews Feb 2027 — High Risk/Priority): * Status: 4.82M pages (up 3x in 12 months); paying $3,200/mo ($38.4k/yr) against $5,784 cost. Requested a 2-year lock-in at the current rate. * Play: Flatly decline the 2-year rate freeze. Present their 3x volume growth and offer an enterprise tier at $10,640/mo ($127.7k ARR) ($1k base + 4.82M pages @ $0.0020). Demonstrate that DocuLens would charge them $43,060/mo ($516k ARR). Our proposal saves them $388k annually versus competitors. 3. Arcadia Health (Renews Mar 2027 — Retention Risk): * Status: 3.31M pages, paying $4,500/mo ($54k ARR); currently evaluating competitors. * Play: Competitive intelligence shows Arcadia cannot leave for a lower price: ExtractIQ would charge them $49,650/mo and DocuLens $30,980/mo. Offer Arcadia a multi-year deal with a grandfathered volume tier at $7,620/mo ($91.4k ARR) ($1k base + 3.31M pages @ $0.0020). They get price predictability; we move gross margin from 11.7% to 47.9%. 4. Halden Insurance (Jun), Meridian Claims (Sep), Brightwater Bank (Nov): * Play: Conduct mid-term Quarterly Business Reviews (QBRs) 90 days before renewal. Share data on their volume growth, introduce the new enterprise volume-tier framework, and negotiate smooth transitions without surprising client procurement teams.

Summary Recommendation

Tom's flat 30% increase damages our acquisition funnel, fails to make heavy users profitable, and risks enterprise litigation or churn. By shifting to a consumption-aligned hybrid model, Parsewell protects its developer brand, doubles monthly revenue to $82.9k, and achieves a sustainable 60.3% gross margin that expands as underlying inference costs drop.

Claims the judge couldn’t find in the brief

  • The test ran for 13 days during the mid-December holiday slowdown.

Grades and run details

Decision model 65 · LLM judge 50
Decision model checks
  • passUses the supplied evidence correctly7%
  • partialAddresses the actual decision76%
  • partialRespects explicit constraints48%
  • failIdentifies material uncertainty61%
  • partialAvoids unsupported claims16%
  • passProduces the required deliverable31%
  • passFixes the value metric100%
  • passDoesn't trust the price test100%
  • passPlans the renewals100%
  • passEach tier for a different buyer37%
  • failGets the base of every number right55%
  • partialPriced on willingness to pay52%
  • partialMoves existing customers fairly86%
Run
Run
#1
API response time
1.6 min
Submitted
6 Oct 2026

Usable with a quick edit12 of 13 checks passed

The output is a complete, evidence-backed memo that a PM could finalize with light edits, as all reasoning is clear and no fact-checking or structural rework is needed.

Output

To: CEO From: Head of Product Re: Alternative to a 30% January increase

Recommendation

Don't apply a flat 30%. Move pricing from seats with unlimited pages to a model that charges for pages. Self-serve keeps seats but adds an included page allowance and a per-page overage. Enterprise moves to a committed per-page rate at renewal. At current usage, this takes gross margin from 24% to about 53%.

Tom's increase would not get us to 41%. That figure assumes all contracts reprice in January, and only one enterprise contract renews then. Self-serve-only in January gives about 34%.

What the data says

  • The margin problem is enterprise. Six contracts account for 71% of pages (19.4M of 27.4M) and 71% of serving cost ($23.2k of $32.9k a month). They pay $19.0k, so they lose $4.2k a month (−22%). Four of six are negative.
  • Enterprise pays too little per page. Effective rates run $0.0007 to $0.0015 per page against a serving cost of $0.0012. Seat pricing lets page volume grow without any increase in what we charge. Northwind's pages tripled in a year and its bill didn't move.
  • Self-serve is healthy. It runs at 60% margin. Startups are at 87%, and mid-market is at 40%, dragged down by heavy accounts. Q and P are negative.
  • We are far below market. DocuLens lists at $0.012 per page up to 1M and $0.008 above, plus a $500 platform fee. At that list price, Northwind's 4.8M pages would cost about $43k a month, against our $3.2k. Enterprise discounts narrow this gap, but not by much.

Proposed model

  • Self-serve: $99 per seat per month, including 25,000 pages per seat. $0.002 per page above that. Seats stay because they are simple and match Formwise. Our overage rate is one-fifth of Formwise's $0.01.
  • Enterprise: a committed monthly page volume at $0.002 per page, with overage at the same rate. No seat fee. Offered on two-year terms with a price lock.
  • Floor: $0.0017 per page. Anything below it needs CEO sign-off.

Because serving cost is falling about 10% a quarter, a locked rate becomes more profitable each quarter. If that holds, enterprise margin reaches about 60% within a year at current volume.

Effect by segment

Monthly run-rate at current usage, before churn, at $0.0012 per page.

SegmentAccounts (seats / pages per month)TodayProposedRevenue per monthGross margin
Startups (self-serve)50 (101 / 1.09M)$99/seat, unlimited pages$99/seat incl. 25k pages; $0.002 above$10.0k → $10.1k (+1%)87% → 87%
Mid-market (self-serve)24 (142 / 6.97M)$99/seat, unlimited pages$99/seat incl. 25k pages; $0.002 above$14.1k → $21.0k (+49%)40% → 60%
Enterprise6 (65 / 19.4M)Negotiated per seat, unlimited pages$0.002 per committed page, no seat fee$19.0k → $38.7k (+104%)−22% → 40%
Total80$43.1k → $69.8k (+62%)24% → 53%

Gross margin dollars rise from $10.2k to $36.9k a month. Startups barely change because most use less than 25k pages per seat. Mid-market carries the most self-serve risk: a few heavy accounts see large increases, such as Q ($594 to $1,382) and T ($792 to $1,498).

Getting there

Self-serve. Announce in January and take effect February 1 with 30 days' notice. New sign-ups get the new model immediately. For existing heavy mid-market accounts, cap the increase at 50% for the first three months. Offer annual prepay as an option.

Enterprise, by renewal date. No contract is repriced mid-term.

  • Copperfield (January 2027): $0.002 per page takes it from $2.4k to about $4.8k a month. This is the first test of the model.
  • Northwind (February 2027): Its ask is "two years at the same rate." At the same dollar amount, the effective rate would fall to about $0.0007 per page while volume triples. Counter with a two-year term, a committed 4.8M pages a month, $0.0017 in year one, and $0.002 in year two. That is about $8.2k a month in year one. Trade the rate for term and commitment. If they refuse to go below $0.0017, let them walk. They lose us $2.6k a month today, and losing them would improve gross margin dollars.
  • Arcadia (March 2027): It is evaluating a competitor and is the only enterprise account with positive margin ($528 a month). Offer $0.002, benchmark against ExtractIQ at their volume, and concede only down to the floor and only on a multi-year deal.
  • Halden (June), Meridian (September), Brightwater (November): standard $0.002.

The price test

The test's B arm was $129 against $99, which is a 30% increase. That makes it the most relevant evidence we have, and it doesn't support the January increase.

  • The result is inconclusive. Conversion was 5.0% against 4.8%. The 95% interval on the difference is roughly −1.7 to +2.1 points. That interval can't rule out losing a third of conversion. "Not significant" does not mean "no effect."
  • The assignment looks broken. The arms received 1,240 and 870 visitors against a 50/50 design. That is a sample-ratio mismatch (chi-square ≈ 65). Until someone explains it, we can't trust either arm.
  • Its scope is narrow. It ran two weeks over the holidays, measured new sign-ups only, and counted first payments only. It says nothing about existing customers, seat expansion, churn, or heavy page users.

We shouldn't cite it as cover. Rerun it properly: a six-week randomized 50/50 test of the new model on new sign-ups, plus a notice-based read on existing self-serve accounts.

Why not 30% across the board

  • It doesn't fix enterprise. A 30% increase moves enterprise from −22% to about 6% margin. Even then, it can't be applied mid-term.
  • Seats don't track cost. Serving cost is about $13 per seat for startups and about $480 per seat for Northwind. A per-seat increase raises both sides equally and leaves the mismatch in place.
  • It taxes the wrong customers. Light users absorb the same percentage increase as heavy users, while the heavy users are the ones we under-recover on.

Risks

  • Enterprise pushback. A roughly 2x jump at renewal is the real risk. Expect one or two losses. The two-year lock and staggered renewal calendar limit the damage.
  • Cost decline stalls. If serving cost stops falling, the $0.0017 floor needs review each quarter.
  • Usage shifts. Heavy self-serve users may trim usage to avoid overage. That is acceptable given the margin, but we should watch conversion closely.

Decisions requested

  1. Reject the flat 30% increase.
  2. Approve the self-serve model effective February 1, with a 90-day cap for heavy mid-market accounts.
  3. Approve the enterprise per-page rate card, with a $0.0017 floor and CEO sign-off required below it.
  4. Rerun the price test properly before any new-signup price change.

Grades and run details

Decision model 81 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly21%
  • passAddresses the actual decision33%
  • passRespects explicit constraints30%
  • passIdentifies material uncertainty76%
  • passAvoids unsupported claims23%
  • passProduces the required deliverable45%
  • passFixes the value metric100%
  • passDoesn't trust the price test100%
  • passPlans the renewals100%
  • partialEach tier for a different buyer7%
  • failGets the base of every number right46%
  • partialPriced on willingness to pay33%
  • partialMoves existing customers fairly61%
Run
Run
#1
API response time
1.9 min
Submitted
8 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 8

Uses the supplied evidence correctlyWrongMixedRight
Gemini 4 Argon · API

The current-state revenue and cost numbers for mid-market and startups in the table and diagnosis do not match the CSV totals, making several factual claims unsupported.

Gemini 3.8 Flash · API

The claim that the price test ran for 13 days is not supported; the source says 8 to 21 December, which is 14 days.

Haiku 5.5 · API

All factual claims about the current situation, including the enterprise loss, page shares, margins, renewal dates, and test details, are directly drawn from the supplied data or correct arithmetic.

Addresses the actual decisionRightWrongRight
Gemini 4 Argon · API

The output commits clearly to an alternative usage-based pricing model and addresses the CEO with a call.

Gemini 3.8 Flash · API

The output does not state what result or condition would change the recommendation.

Haiku 5.5 · API

The output commits early and unambiguously to rejecting the 30% flat rise and proposes a specific alternative pricing model, framed for the CEO, with conditions that would change the call.

Identifies material uncertaintyWrongWrongRight
Gemini 4 Argon · API

The output does not name specific unknowns that could change the pricing decision (e.g., true customer WTP, churn risks) or state what would resolve them.

Gemini 3.8 Flash · API

The output does not name specific unknowns that could change the decision or how they would be resolved.

Haiku 5.5 · API

The output names unknowns such as enterprise pushback, cost decline stalling, usage shifts, and the price test's broken assignment, and says how they would be resolved or monitored.

Avoids unsupported claimsRightWrongRight
Gemini 4 Argon · API

The main claims about causes and forecasts are presented as analysis, not as unsupported fact; the factual errors are covered under evidence.

Gemini 3.8 Flash · API

Forecasts like 'doubles monthly revenue to $82.9k, and achieves a sustainable 60.3% gross margin' are presented as confident claims without being labelled as projections.

Haiku 5.5 · API

Interpretations like 'the margin problem is enterprise' are directly backed by the arithmetic, and forecasts are conditioned on assumptions such as cost decline continuing.

Fixes the value metricMixedRightRight
Gemini 4 Argon · API

The output shows enterprise accounts are loss-making but does not compute the enterprise share of total pages (e.g., 19.4M of 27.4M) as the strong answer requires.

Gemini 3.8 Flash · API

The output computes enterprise share of pages and negative margin, and proposes a hybrid usage-based model with per-page tiers.

Haiku 5.5 · API

The output computes enterprise's 71% page share and negative margin, then proposes moving to usage-based pricing with per-page charges for both self-serve overage and enterprise.

Gets the base of every number rightWrongMixedMixed
Gemini 4 Argon · API

The current-month revenue and cost sums in the table are wrong, so derived percentages and margins are unreliable.

Gemini 3.8 Flash · API

All derived percentages and margins check out against the supplied data; the 13-day error is a factual claim, not a computation error.

Haiku 5.5 · API

Derived figures like segment revenues, margins, and percentage increases check out against the usage data, with bases clearly stated and step changes correctly computed.

Priced on willingness to payWrongWrongRight
Gemini 4 Argon · API

Prices are set relative to competitors and cost rather than evidence of what each segment would pay; no willingness-to-pay data is used.

Gemini 3.8 Flash · API

The output relies on competitor pricing as a proxy for willingness to pay but does not state where the evidence is thin or use direct WTP signals from the material.

Haiku 5.5 · API

Prices are set well below competitor market rates (e.g., $0.002 vs DocuLens $0.008-$0.012) and justified by the value gap and existing margin recovery, with explicit admission that enterprise pushback is a risk.

Moves existing customers fairlyRightWrongRight
Gemini 4 Argon · API

It proposes new-signups first, a 90-day transition for affected self-serve customers, and a tailored playbook for enterprise renewals.

Gemini 3.8 Flash · API

The transition plan lacks a mechanism for watching churn, which is required for moving existing customers fairly.

Haiku 5.5 · API

Existing self-serve accounts get 30 days' notice, a 50% cap for heavy mid-market accounts for three months, and enterprise transitions only at renewal, with churn risk explicitly noted.

All got right 5

Respects explicit constraintsRightRightRight
Gemini 4 Argon · API

The response is under 1,300 words, includes the required table, price-test assessment, and enterprise renewal plan.

Gemini 3.8 Flash · API

The output is a memo to the CEO, includes the required table, and appears to be within 1,300 words.

Haiku 5.5 · API

The output is under 1,300 words, includes the required table of the model and effect by segment, addresses the CEO, and prohibits the flat 30% rise.

Produces the required deliverableRightRightRight
Gemini 4 Argon · API

A complete memo to the CEO with the requested table, segment impact, rollout, and price-test critique is provided.

Gemini 3.8 Flash · API

The memo with table is complete, addressed to the CEO, and provides an actionable plan.

Haiku 5.5 · API

The memo is in correct form for the CEO, contains the required elements (new model, table, effect, enterprise renewals, price test view), and is ready to act on.

Doesn't trust the price testRightRightRight
Gemini 4 Argon · API

It identifies the sample ratio mismatch and underpowered/holiday flaws, and does not rest the pricing decision on the test.

Gemini 3.8 Flash · API

The output names the sample ratio mismatch and the holiday timing, and does not rest the decision on the test.

Haiku 5.5 · API

The output identifies the sample ratio mismatch (1240 vs 870 against a 50/50 design), the two-week holiday timing, and the inability to detect a meaningful drop, and refuses to rely on the test.

Plans the renewalsRightRightRight
Gemini 4 Argon · API

Copperfield (Jan), Northwind (Feb, tripled volume), and Arcadia (Mar, evaluating competitor) are each given a specific renewal approach using their dates and notes.

Gemini 3.8 Flash · API

The output gives specific renewal approaches for Copperfield (Jan), Northwind (Feb), and Arcadia (Mar), using their dates and notes.

Haiku 5.5 · API

It handles each near-term renewal: Copperfield in January with a per-page rate, Northwind in February with a two-year escalating rate and walk-away floor, and Arcadia in March with a competitive benchmark and concession limit.

Each tier for a different buyerRightRightRight
Gemini 4 Argon · API

Tiers (Starter, Growth, Enterprise) map to startups, mid-market, and enterprises, each with a different page allowance and buyer value.

Gemini 3.8 Flash · API

The two tiers (Self-Serve and Enterprise) name distinct buyers and include features each values (e.g., SLA, custom models for enterprise).

Haiku 5.5 · API

Self-serve targets occasional users with a seat + included pages model, while enterprise targets high-volume buyers with pure per-page committed volume and no seat fee, each priced on distinct value.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 85% of 68 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Opus 5.5withClaude68.1100.02None
2GPT-6 AstrawithChatGPT70.296.42None
3Sonnet 5.5withAPI69.989.32None
4GPT-6.1 SolwithAPI74.088.72None
5Haiku 5.5withAPI73.776.92None
6GPT-6 LunawithAPI78.464.02None
7Gemini 4 ArgonwithAPI47.659.32None
8Gemini 3.8 FlashwithAPI55.644.22None
9Gemini 3.5 Flash-LitewithGemini41.840.72None

What you get for the cost

GPT-6.1 Sol · API scores within 5 points of Opus 5.5 · Claude at a sixth of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreOpus 5.5 · Claude: task score 85, 122× the cheapestGPT-6 Astra · ChatGPT: task score 84, 108× the cheapestSonnet 5.5 · API: task score 81, 61× the cheapestGPT-6.1 Sol · API: task score 80, 20× the cheapestHaiku 5.5 · API: task score 74, 2.9× the cheapestGPT-6 Luna · API: task score 70, the cheapestGemini 4 Argon · API: task score 52, 15× the cheapestGemini 3.8 Flash · API: task score 48, 9.1× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 40, 2.3× the cheapestOpus 5.5 · ClaudeGPT-6 Astra · ChatGPTSonnet 5.5 · APIGPT-6.1 Sol · APIHaiku 5.5 · APIGPT-6 Luna · APIGemini 4 Argon · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and cost
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API701.0×
Gemini 3.5 Flash-Lite · Gemini402.3×
Haiku 5.5 · API742.9×
Gemini 3.8 Flash · API489.1×
Gemini 4 Argon · API5215×
GPT-6.1 Sol · API8020×
Sonnet 5.5 · API8161×
GPT-6 Astra · ChatGPT84108×
Opus 5.5 · Claude85122×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Setting what you charge, for what, and to whom.

Why it matters

Most products are priced once and never revisited, on a metric that doesn't track value. The fix is rarely a flat rise.

What good looks like

  • Tiers built for distinct buyers
  • A value metric that tracks what customers get
  • Grounded in willingness to pay and cost to serve
  • A plan for existing customers

Deliberately not measured

  • Billing implementation
  • Legal terms
Capability tested

Pricing judgement

The failure we’re looking for

A flat price rise on the wrong value metric

Grading

Decision model and LLM judge, calibrated against a blind PM review