Tasks / Strategy

Price and package a product

Can the model price on the value each kind of customer gets, using real usage and willingness-to-pay data, and get existing customers there?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 56% were usable with at most a quick edit.

Reliably right

  1. Doesn't trust the price test100% pass
    Names the 59/41 split mismatch, holiday timing, and small sample; does not rest the decision on the test.
    Opus 5.5 · Claude · Unlimited pages, falling margins
  2. Plans the renewals100% pass
    Addresses each enterprise renewal with its date and note, including Copperfield in January, Northwind's tripled volume, and Arcadia's competitor evaluation.
    Opus 5.5 · Claude · Unlimited pages, falling margins
  3. Each tier for a different buyer94% pass
    Each tier names its buyer (startups, mid-market, enterprise) and is priced on distinct value: startups protected, mid-market overage on heavy users, enterprise per-page with unlimited seats.
    Opus 5.5 · Claude · Unlimited pages, falling margins

Where it slips

  1. Uses the supplied evidence correctly34% pass
    Total MRR is stated as $11,670 (actual $11,820), and median 'getting expensive' for small agencies is given as about $37 (actual $33) and for agencies as about $55 (actual $51).
    Haiku 5.5 · API · Fifteen dollars for everyone
  2. Gets the base of every number right41% pass
    MRR total and two segment‑level median getting‑expensive figures are incorrectly computed from the supplied data.
    Haiku 5.5 · API · Fifteen dollars for everyone
  3. Avoids unsupported claims52% pass
    Median getting‑expensive values and total MRR are presented as fact but are not supported by the data; these are facts about the current situation, not labelled as assumptions.
    Haiku 5.5 · API · Fifteen dollars for everyone

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Each tier for a different buyer

    Is each plan built for a distinct kind of buyer, with what that buyer values, rather than more-of-the-same at a higher price?

    Passes when Every tier names its buyer and the value it's priced on, and includes what that buyer needs and the lower tier's buyer doesn't.

  2. Priced on willingness to pay

    Are the proposed prices grounded in evidence of what each kind of customer would pay and the value they get, rather than in cost-plus, competitors alone or a round-number rise?

    Passes when Uses the willingness-to-pay and value evidence in the material to set each price, and says where the evidence is thin.

  3. Moves existing customers fairly

    Does the proposal say how existing customers move to the new pricing (who changes first, with what notice, and the value that justifies it)?

    Passes when A transition plan: new customers first or a set notice period for existing ones, tied to value delivered, with how churn will be watched.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for pricing at Inkwell. Our CEO, Priya Nair, wants to move everyone to $19 a user from January (her note is below). Before she announces it, she's asked you for a pricing and packaging proposal: what we charge, for what, to whom, and how we move existing customers onto it. No more than 900 words, with a table of the plans. The files are attached. The account export is a random sample of 150 paying accounts.

What the model was given6 items: About Inkwell, ceo_note.md, accounts_sample.csv, willingness_to_pay_survey.csv (price per user per month at which it's too cheap to trust, a bargain, getting expensive, too expensive), win_loss_last_two_quarters.csv, trial_funnel_q3.md
About InkwellProposal and contract software: write a proposal from templates, send it, get it e-signed. Sold to freelancers and agencies. One plan today: $15 per user per month after a 14-day free trial, with everything included.
ceo_note.md1 lines · Download
From Priya (CEO): "We haven't touched price since launch. Let's go to $19 a user from January for everyone, new and existing. Simple, and a 27% revenue bump."
accounts_sample.csv150 rows · Download
account_id,segment,seats,proposals_sent_90d,proposals_signed_90d,signed_value_90d_usd,months_as_customer,mrr_usd
INK-3001,Small agency (2-10),3,29,14,108444,18,45
INK-3002,Freelancer,1,12,5,5350,7,15
INK-3003,Small agency (2-10),3,62,34,200396,28,45
INK-3004,Agency (11-50),16,139,74,1587596,29,240
INK-3005,Small agency (2-10),6,18,9,42498,30,90
INK-3006,Agency (11-50),29,236,89,1192333,18,435
…
willingness_to_pay_survey.csv (price per user per month at which it's too cheap to trust, a bargain, getting expensive, too expensive)response_id,segment,too_cheap_usd,a_bargain_usd,getting_expensive_usd,too_expensive_usd W400,Freelancer,4,7,14,25 W401,Freelancer,4,9,16,30 W402,Freelancer,5,7,19,20 W403,Freelancer,3,9,14,23 W404,Small agency (2-10),9,12,29,56 W405,Small agency (2-10),10,13,28,47 W406,Small agency (2-10),6,14,31,47 W407,Freelancer,4,10,15,29 W408,Small agency (2-10),6,15,40,56 W409,Small agency (2-10),8,14,40,46 W410,Freelancer,4,8,15,25 W411,Freelancer,4,8,15,19 W412,Freelancer,5,9,18,18 W413,Small agency (2-10),9,12,38,49 W414,Agency (11-50),14,22,49,60 W415,Freelancer,3,6,18,18 W416,Freelancer,4,8,15,28 W417,Small agency (2-10),10,15,37,46 W418,Agency (11-50),13,30,55,73 W419,Freelancer,4,9,15,29 W420,Freelancer,3,7,13,29 W421,Freelancer,4,9,19,21 W422,Agency (11-50),11,29,61,67 W423,Small agency (2-10),8,13,26,62 W424,Freelancer,4,9,20,21 W425,Agency (11-50),10,30,51,73 W426,Small agency (2-10),9,13,38,49 W427,Small agency (2-10),6,17,31,49 W428,Freelancer,4,6,20,28 W429,Small agency (2-10),9,12,41,52 W430,Agency (11-50),9,26,50,69 W431,Freelancer,5,9,15,24 W432,Freelancer,3,8,19,28 W433,Agency (11-50),14,20,66,94 W434,Freelancer,3,10,15,27 W435,Small agency (2-10),7,16,28,46 W436,Freelancer,4,9,17,28 W437,Small agency (2-10),6,18,31,56 W438,Agency (11-50),14,27,59,86 W439,Freelancer,5,10,19,28 W440,Freelancer,5,7,12,25 W441,Agency (11-50),14,23,42,66 W442,Freelancer,4,6,13,26 W443,Small agency (2-10),9,13,34,47 W444,Freelancer,4,10,18,29 W445,Freelancer,4,9,19,29 W446,Small agency (2-10),10,18,30,55 W447,Freelancer,4,10,15,24 W448,Small agency (2-10),10,19,26,40 W449,Small agency (2-10),8,15,41,42 W450,Freelancer,5,8,14,26 W451,Freelancer,4,6,14,30 W452,Small agency (2-10),7,18,37,43 W453,Small agency (2-10),7,19,27,42 W454,Freelancer,3,8,13,29 W455,Agency (11-50),10,24,57,99 W456,Freelancer,4,9,17,24 W457,Small agency (2-10),9,17,32,55 W458,Freelancer,4,9,14,23 W459,Small agency (2-10),9,15,30,38 W460,Freelancer,4,7,18,28 W461,Freelancer,4,8,14,23 W462,Small agency (2-10),9,17,27,53 W463,Freelancer,3,9,14,21 W464,Agency (11-50),13,31,58,63 W465,Agency (11-50),15,26,42,80 W466,Freelancer,5,9,18,19 W467,Small agency (2-10),9,15,27,49 W468,Freelancer,4,8,14,23 W469,Small agency (2-10),9,12,29,51 W470,Agency (11-50),14,26,51,92 W471,Freelancer,5,7,14,26 W472,Freelancer,4,9,18,23 W473,Freelancer,3,10,12,22 W474,Freelancer,5,8,16,29 W475,Agency (11-50),11,20,67,62 W476,Agency (11-50),13,28,49,79 W477,Freelancer,5,10,14,22 W478,Agency (11-50),11,21,50,91 W479,Freelancer,4,9,13,29 W480,Freelancer,3,6,15,25 W481,Small agency (2-10),9,16,29,53 W482,Freelancer,4,10,18,27 W483,Small agency (2-10),10,15,40,54 W484,Agency (11-50),11,20,42,80 W485,Small agency (2-10),8,14,42,52 W486,Freelancer,5,8,16,30 W487,Small agency (2-10),8,15,27,49 W488,Agency (11-50),15,26,58,70 W489,Small agency (2-10),8,19,26,51 W490,Small agency (2-10),6,14,26,42 W491,Small agency (2-10),7,16,37,45 W492,Freelancer,4,6,13,21 W493,Small agency (2-10),8,18,42,44 W494,Small agency (2-10),8,17,38,52 W495,Small agency (2-10),7,13,26,45 W496,Small agency (2-10),9,18,42,56 W497,Freelancer,5,8,17,28 W498,Agency (11-50),10,24,51,92 W499,Freelancer,4,6,19,26 W500,Agency (11-50),10,21,58,93 W501,Freelancer,3,6,15,20 W502,Freelancer,5,7,19,20 W503,Small agency (2-10),9,15,40,59 W504,Small agency (2-10),7,19,29,58 W505,Small agency (2-10),8,18,27,62 W506,Freelancer,5,7,14,26 W507,Small agency (2-10),8,14,40,46 W508,Freelancer,5,9,13,28 W509,Freelancer,4,10,12,25 W510,Freelancer,4,10,18,28 W511,Freelancer,4,10,14,20 W512,Agency (11-50),10,20,42,73 W513,Freelancer,5,9,14,30 W514,Freelancer,4,7,15,26 W515,Freelancer,3,9,15,27 W516,Freelancer,4,9,16,19 W517,Small agency (2-10),8,13,36,54 W518,Freelancer,5,10,12,19 W519,Freelancer,3,8,19,24
win_loss_last_two_quarters.csv160 rows · Download
deal_id,quarter,segment,outcome,price_objection,lost_to
D-7001,2026-Q3,Small agency (2-10),won,no,
D-7002,2026-Q2,Freelancer,no decision,no,stayed with current process
D-7003,2026-Q3,Small agency (2-10),won,no,
D-7004,2026-Q2,Small agency (2-10),lost,no,PandaDoc
D-7005,2026-Q3,Agency (11-50),no decision,no,stayed with current process
D-7006,2026-Q2,Small agency (2-10),no decision,no,stayed with current process
…
trial_funnel_q3.md9 lines · Download
# Trial funnel, Q3 2026 (14-day free trial, then $15 per user per month)

| Segment | Trials started | Became paying | Paying accounts that cancelled in their second month |
|---|---|---|---|
| Freelancer | 1,900 | 171 | 38% |
| Small agency (2-10) | 420 | 130 | 6% |
| Agency (11-50) | 60 | 22 | 0% |
…
What a strong answer doesThe answer key the graders mark against

Proposes tiers built around three different buyers, not one price. Agencies close about $20,000 a seat each month through Inkwell (small agencies about $8,000, freelancers under $2,000), use the shared library, approvals and CRM sync, and put 'getting expensive' at about $51 a user (small agencies about $31), so they are badly underpriced at $15; win rates are high (125 of 160 deals) and price objections rare (6 of 160), mostly from freelancers. Freelancers put 'getting expensive' at about $15 and 'too expensive' at about $26, convert at 9% and cancel at 38% in month two, so $19 would push them past their limit and lose the few who stay. So: a freelancer plan at or below today's price (or a free plan with a usage limit, such as a few proposals a month, rather than a time-limited trial), a team plan for small agencies around the low $20s to $30, and an agency plan in the $40s or with the features agencies value (library, approvals, CRM sync), named for the buyer. Rolls it out to new customers first and gives existing customers notice and a grace period tied to new value, and says how it will measure the change (conversion, churn, revenue per account).

Critical failures (cap the score at 40)
  • Recommends a single flat price for every customer
Task

Mid-level · v1.1

The outputs

8 setups answered this brief.

Needs more than a quick edit3 of 12 checks passed

Contains a factual error in the median too-expensive claim and omits churn monitoring and value-gap analysis from account data, requiring rework.

Output

MEMORANDUM

TO: Priya Nair, CEO FROM: Pricing PM, Inkwell DATE: October 24, 2026 SUBJECT: Q1 2027 Pricing & Packaging Proposal

---

Executive Summary

Your instinct to capture more value is spot on—we haven’t touched price since launch, and our growth gives us pricing power. However, moving everyone to a flat $19/user in January carries hidden risks.

Based on our Van Westendorp pricing survey, win/loss data, and trial cohort metrics, a flat per-user hike across all segments will cause churn spikes among freelancers (whose median "too expensive" threshold is $25/mo) and leave money on the table for larger agencies (who derive immense value from collaboration features and are willing to pay significantly more).

Instead of a blunt instrument, I propose a packaged tier strategy starting January 1st that protects our low-end acquisition funnel, extracts enterprise-grade value from larger agencies, and grandfathering existing customers gracefully.

---

Data Insights & Rationale

1. Willingness to Pay Varies Widely by Segment: * Freelancers ($15 current): Median "too expensive" point is $25–$28. A jump to $19 is survivable, but pushing past $20 risks driving them back to free e-signature tools and Google Docs (our #1 lost-to competitor). * Small Agencies (2–10 seats): Median "too expensive" point is $46–$55 ($15–$18/user effective). They heavily utilize team features (shared libraries, approval workflows). * Agencies (11–50 seats): Median "too expensive" point is $70–$93 ($6–$8/user effective at scale). They view Inkwell as core operations infrastructure and integrate it with their CRM. They are severely under-monitored on a flat per-user model.

2. The Freelancer Churn Trap: Our Q3 trial funnel shows that 38% of freelancers cancel in their second month. Raising their price immediately in January without added value will accelerate this churn.

---

Proposed Pricing & Packaging Structure (Effective January 1)

We will transition from a single monolithic plan to a 3-tier product-led packaging model.

Plan NameTarget SegmentPriceCore Packaging & Features
:---:---:---:---
StarterFreelancers (1 user)$19 / mo (Flat)1 user, unlimited proposals, basic templates, e-sign, standard branding.
GrowthSmall Agencies (2–10 seats)$25 / user / moStarter features + Shared content library, team approval workflows, custom branding, priority support.
ScaleAgencies (11–50 seats)$39 / user / moGrowth features + CRM integrations (HubSpot/Salesforce), advanced audit logs, dedicated account review, SSO.

---

Revenue Impact Projections

  • Starter Tier ($19 flat): Retains freelancers at your target price point while securing a 26% ARPU lift on single-user accounts.
  • Growth & Scale Tiers: By packaging high-value features (CRM sync, approvals) into higher tiers, we capture enterprise willingness-to-pay. For an average 20-seat agency currently paying $300/mo ($15/seat), moving to the Scale tier at $39/seat shifts their MRR to $780—a 160% expansion, rather than a modest 27% bump.

---

Migration & Rollout Plan for Existing Customers

To avoid churn and backlash while hitting our January revenue targets, we will execute the following transition:

1. Grandfathering Window (Peace of Mind): Existing customers will be grandfathered on their current $15/user rate for 90 days (through March 31, 2027), provided they do not change plans. This prevents a sudden holiday-season churn spike. 2. Value-Based Migration (April 1): * Freelancers automatically transition to the Starter plan ($19/mo flat) on April 1, with a 30-day advance notice emphasizing new template updates. * Agencies will be mapped to the Growth or Scale tiers based on their current feature usage (e.g., accounts currently utilizing CRM sync will be slotted into Scale, with a personalized outreach from success reps highlighting the added utility). 3. Opt-Out & Retention Safety Net: Any existing customer who receives the April price change notice will be offered an annual prepayment discount (2 months free) to lock in a intermediate rate, preserving cash flow while securing retention.

Next Steps

With your approval on this structure, product and marketing will finalize the feature gating by November 15, allowing us to build billing migrations and draft customer communications well ahead of the January launch.

Claims the judge couldn’t find in the brief

  • Freelancers' median 'too expensive' threshold is $25/mo.

Grades and run details

Decision model 42 · LLM judge 38
Decision model checks
  • failUses the supplied evidence correctly70%
  • partialAddresses the actual decision34%
  • failRespects explicit constraints26%
  • failIdentifies material uncertainty63%
  • failAvoids unsupported claims59%
  • partialProduces the required deliverable35%
  • partialSees who's underpriced66%
  • passAnswers the CEO's flat rise22%
  • passEach tier for a different buyer80%
  • failGets the base of every number right69%
  • partialPriced on willingness to pay88%
  • passMoves existing customers fairly61%
Run
Run
#1
Time to output
5 s
Submitted
5 Oct 2026

Needs more than a quick edit5 of 12 checks passed

Factual errors in the claims (pricing objections, WTP thresholds) mean a PM would need to re-verify the evidence before using the proposal.

Output

MEMORANDUM

TO: Priya Nair, CEO FROM: Pricing PM DATE: October 24, 2026 SUBJECT: Pricing & Packaging Proposal: Value-Based Tiering vs. Flat $19 Increase

---

Executive Summary

A blanket increase to $19/user/month appears simple, but the data indicates it will harm growth. A flat $19 will accelerate churn among freelancers (who already churn at 38% in month 2) while leaving massive revenue on the table with agencies, who consider $19 suspiciously cheap and are willing to pay $35–$59/user.

Instead of a single price hike, we should introduce a three-tier packaging model aligned with willingness-to-pay (WTP), feature utilization, and economic value delivered. This structure protects top-of-funnel conversion, monetizes high-value agency workflows, and delivers an estimated 65–85% ARR increase (vs. ~27% under the flat $19 plan).

---

Key Findings from Account, Survey, and Funnel Data

1. Freelancers are extremely price-sensitive and churn-prone: * Funnel: 1,900 trials yielded 171 customers (9.0% conversion), with 38% cancelling in Month 2. * Win/Loss: All pricing objections in Q2/Q3 came from freelancers losing to free alternatives (Google Docs + free e-sign). * WTP Data: Freelancer median "getting expensive" threshold is $15.00; "too expensive" begins at $22.00. Moving them to $19 pushes them into the danger zone, exacerbating our 38% churn. * Usage: Single seat, 1–2 templates, signing ~$5k/quarter.

2. Agencies are vastly underpaying relative to value: * Small Agencies (2–10 seats): Trial-to-paid is 31.0%, Month 2 churn is only 6%. WTP survey shows "bargain" up to $19, "getting expensive" at $31–$40, and "too expensive" at $49–$56. They sign $100k+ in client contract value per quarter. * Mid-Sized Agencies (11–50 seats): Trial-to-paid is 36.7%, Month 2 churn is 0%. In WTP surveys, several agencies rated $14–$15 as "too cheap to trust", with acceptable pricing running $49–$66/seat. They manage millions in signed pipeline and rely on workflow governance. * Feature Divergence: Agencies actively use CRM sync, shared asset libraries, and internal approval workflows—enterprise features currently given away for $15. When we lose agencies, it is to PandaDoc on feature depth, never on price.

---

Proposed Pricing & Packaging

We replace our single $15 plan with three distinct tiers:

TierStarter (Freelancer)Growth (Small Agency)Scale (Agency 11–50)
Target AudienceSolo operators, freelancersBoutique agencies (2–10 seats)Growing agencies & studios (10+ seats)
Price (Monthly)$15 / user / mo$29 / user / mo$49 / user / mo
Price (Annual - 20% off)$12 / user / mo ($144 billed annually)$24 / user / mo ($288 billed annually)$39 / user / mo ($468 billed annually)
Seat Limits1 seat maximumUp to 10 seats (min. 2 seats)10+ seats (min. 5 seats)
Core Features• Unlimited proposals & e-signatures<br>• Standard template library<br>• Client view tracking<br>• Email supportEverything in Starter, plus:<br>• Shared team content/asset library<br>• Basic CRM integration (HubSpot, Pipedrive)<br>• Custom branding & subdomains<br>• Role-based permissionsEverything in Growth, plus:<br>• Multi-step internal approval workflows<br>• Advanced CRM sync (Salesforce, custom fields)<br>• Template locking & legal guardrails<br>• Priority onboarding & dedicated CSM
Trial Model14-day trial of Growth tier14-day trial of Growth tierDemo-led / Sales-assisted trial

#### Why this structure works: * Starter ($15/mo): Defends the baseline against free alternatives. Annual billing ($12/mo) locks in cash upfront and directly attacks the 38% Month 2 churn. * Growth ($29/mo): Sits directly in the sweet spot for 2–10 person teams ($26–$34 WTP). It monetizes shared libraries and CRM integrations. * Scale ($49/mo): Captures willing enterprise budget, positions us against PandaDoc, and packages approval workflows that multi-seat agencies require.

---

Migration & Grandfathering Strategy (Launch: January 1)

Forcing existing clients to an unannounced 27% increase risks immediate churn and brand backlash. We should execute a structured migration playbook:

1. Freelancers (Starter): * Action: No price increase on monthly ($15/mo). * Incentive: Offer an upgrade to Annual at $12/user/month ($144/yr) during December. This improves net cash flow and curbs Month 2 drop-off.

2. Existing Agencies (Small & Mid-Sized): * Grace Period (Grandfathering): Maintain current $15/seat pricing for 6 months (through June 30) on their current feature footprint. * Opt-in Upgrade Campaign: Offer 1 year of the Growth plan at $22/seat/mo or Scale at $39/seat/mo if committed annually before January 15. * Feature Gating: Existing accounts keep existing active templates. Advanced features released in Q1 (advanced approvals, deep CRM sync) will require the Growth or Scale tier.

3. New Signups (Effective Jan 1): * Default straight to new public tiering. Trials default to the Growth tier.

---

Financial Impact Comparison

Based on 150-sample extrapolation (48% Freelancers, 41% Small Agencies, 11% Agencies):

  • CEO Flat $19 Proposal:
  • Freelancer churn will rise ~15–20% due to price sensitivity.
  • Captures only +$4/seat on agencies capable of paying +$14 to +$34/seat.
  • Net ARR growth: ~18–22% after factoring churn.
  • Recommended Tiered Proposal:
  • Retains freelancer volume while improving annual commitments.
  • Blended ARPU expands from $15 to ~$29.50/seat.
  • Net ARR growth: ~74%, while insulating the business from competitive free tools and funding enterprise feature parity against PandaDoc.

I am ready to finalize the in-app messaging, billing logic, and sales enablement collateral upon your approval.

Claims the judge couldn’t find in the brief

  • All pricing objections in Q2/Q3 came from freelancers losing to free alternatives
  • Freelancer 'too expensive' begins at $22.00
  • WTP survey shows small agency 'too expensive' at $49–$56
  • Agency acceptable pricing running $49–$66/seat

Grades and run details

Decision model 63 · LLM judge 38
Decision model checks
  • failUses the supplied evidence correctly56%
  • passAddresses the actual decision25%
  • partialRespects explicit constraints41%
  • failIdentifies material uncertainty78%
  • failAvoids unsupported claims59%
  • passProduces the required deliverable46%
  • passSees who's underpriced80%
  • passAnswers the CEO's flat rise100%
  • passEach tier for a different buyer100%
  • failGets the base of every number right78%
  • passPriced on willingness to pay65%
  • passMoves existing customers fairly90%
Run
Run
#1
API response time
24 s
Submitted
6 Oct 2026

Usable with a quick edit11 of 12 checks passed

The proposal is well-structured and actionable. The omission of signed value per seat is minor and can be added in a quick edit without redoing the reasoning.

Output

Recommendation: raise agency pricing, not every user to $19

Launch segmented plans in January 2027. Keep freelancers at $15 per user; price agencies for the collaboration and CRM features they use. Priya’s 27% figure is the gross increase if every account stays: it does not account for cancellations or lost conversions.

PlanMonthly priceForIncluded
------:------
Solo$15/user (one seat)FreelancersCreate proposals from templates, send them, and collect e-signatures
Team$19/user (2–10 seats)Small agenciesSolo features, plus shared content library and approval workflows
Business$25/user (11+ seats)Larger agenciesTeam features, plus CRM sync

Keep the 14-day free trial. Use seat count as the default plan guide, with sales able to recommend the right fit where an account’s needs differ.

Why this structure

  • Freelancers show the clearest risk from a blanket increase. In Q3, 9% of freelancer trials became paying customers, versus 31% for small agencies and 37% for larger agencies. Of paying freelancers, 38% cancelled in month two, compared with 6% of small agencies and none of the larger agencies. A higher entry price could compound an already weak conversion and retention picture.
  • Survey responses support different price points by segment. Freelancers typically start to feel price pressure around the mid-teens; agencies report much higher thresholds. The proposed $19 Team and $25 Business prices remain well below the survey’s typical “getting expensive” levels for agencies. Treat the survey as directional, not a guarantee of willingness to pay.
  • Agency packaging reflects observed value. Agencies use the shared library, approval workflows, and CRM sync; freelancers mostly use one template and e-signature. The Q3 account sample also shows much heavier proposal activity in agencies. Charging more for agency collaboration and integration is more defensible than charging every freelancer more for features they may not use.
  • Sales evidence argues against relying on price increases alone. Recorded losses include freelancers choosing free tools and agencies choosing PandaDoc; many no-decisions stayed with their current process. Price objections appear, but are not the dominant story in these records.

Move existing customers carefully

  1. New customers: Apply the new plan prices from January 2027.
  2. Existing freelancers: Keep them on Solo at $15. Do not force a price increase or a feature downgrade.
  3. Existing agencies: Give each account its recommended plan, explain the feature mapping, and grandfather its current access for 12 months. Provide at least 60 days’ notice before any price change; for annual customers, change pricing at renewal, not mid-term.
  4. Phase agency increases: Cap each renewal increase at 20%, with a clear schedule to reach the recommended list price within three renewals. Let customers adjust seats or select a lower plan before renewal; do not remove access unexpectedly.
  5. Review before scaling: Track trial conversion, second-month cancellations, agency churn, and downgrade requests by segment. Revisit prices after two quarters rather than assuming the sample or survey predicts the full customer base.

The 150-account export is a random sample, not a forecast. The proposed prices are a testable segment-based starting point; blanket repricing would add avoidable freelancer risk.

Grades and run details

Decision model 75 · LLM judge 92
Decision model checks
  • passUses the supplied evidence correctly23%
  • passAddresses the actual decision42%
  • partialRespects explicit constraints18%
  • partialIdentifies material uncertainty80%
  • partialAvoids unsupported claims33%
  • partialProduces the required deliverable25%
  • partialSees who's underpriced85%
  • passAnswers the CEO's flat rise73%
  • passEach tier for a different buyer91%
  • passGets the base of every number right26%
  • partialPriced on willingness to pay48%
  • passMoves existing customers fairly100%
Run
Run
#1
API response time
25 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 10

Uses the supplied evidence correctlyWrongWrongRight
Gemini 3.5 Flash-Lite · Gemini

States freelancers' median too-expensive as $25/mo, but the survey data gives a median of $26.

Gemini 3.8 Flash · API

Multiple factual claims are incorrect: all pricing objections did not come only from freelancers losing, freelancer 'too expensive' minimum is 18 not 22, and WTP ranges for agencies are misstated.

GPT-6 Luna · API

All factual claims about the current situation are directly supported by the brief or supplied context, with no invented facts.

Addresses the actual decisionWrongMixedRight
Gemini 3.5 Flash-Lite · Gemini

Does not say what result or condition would change the recommendation.

Gemini 3.8 Flash · API

The output does not state what result or condition would change the recommended tiered approach, only commits to it without a clear pivot point.

GPT-6 Luna · API

Commits early to a clear answer (segmented plans with Solo/Team/Business), framed for Priya, and says revisiting after two quarters is the condition to change it.

Respects explicit constraintsMixedRightRight
Gemini 3.5 Flash-Lite · Gemini

Includes a plan table, stays under 900 words, and addresses the CEO as requested.

Gemini 3.8 Flash · API

The output is a pricing and packaging proposal under 900 words with a plan table, respecting the form and length requested.

GPT-6 Luna · API

The output is under 900 words, includes a plan table, and respects the form and reader requested.

Identifies material uncertaintyWrongWrongRight
Gemini 3.5 Flash-Lite · Gemini

Does not name unknowns that could change the decision or how they would be resolved.

Gemini 3.8 Flash · API

No material unknowns are named, and the output does not say how uncertainties (e.g., small agency sample, actual churn impact) would be resolved or could change the decision.

GPT-6 Luna · API

Identifies that the survey is directional and the sample is not a full forecast, and proposes tracking metrics to revisit prices.

Avoids unsupported claimsWrongWrongRight
Gemini 3.5 Flash-Lite · Gemini

Presents forecasts like 'a flat hike will cause churn spikes' as fact without labelling them as hypotheses.

Gemini 3.8 Flash · API

The output presents 'All pricing objections came from freelancers losing' and specific WTP thresholds as established fact when the data does not fully support them.

GPT-6 Luna · API

Interpretations like survey being directional are clearly labeled, and no confident causal claims are made beyond what the evidence supports.

Sees who's underpricedWrongRightWrong
Gemini 3.5 Flash-Lite · Gemini

Does not use signed value per seat from the account sample to show agencies are underpriced relative to value delivered.

Gemini 3.8 Flash · API

The proposal uses signed value per seat and survey price points to show agencies are underpriced and freelancers are near their limit.

GPT-6 Luna · API

Does not use signed value per seat from the account sample to quantify how much more value agencies get; relies on features and activity mentions but omits the explicit value metric.

Answers the CEO's flat riseMixedRightRight
Gemini 3.5 Flash-Lite · Gemini

Does not show that $19 is above freelancers' 'getting expensive' point ($15 median), only references 'too expensive'.

Gemini 3.8 Flash · API

It explains that $19 would push freelancers past their getting-expensive point and still leave agencies far below their willingness to pay.

GPT-6 Luna · API

Explains that $19 would exceed freelancers' price comfort zone (mid-teens) and still be far below agencies' willingness to pay, using trial and survey data.

Gets the base of every number rightMixedWrongRight
Gemini 3.5 Flash-Lite · Gemini

Derived percentages (38% churn, 9% conversion, 160% expansion, 26% lift) are correctly calculated from the data.

Gemini 3.8 Flash · API

The freelancer 'too expensive' floor is misstated as $22 instead of $18, and agency WTP ranges are presented as narrower than the data shows, without stating the base.

GPT-6 Luna · API

All percentages (9%, 31%, 37%, 38%, 6%, 0%, 27%) are correctly computed from the supplied data with the correct denominators.

Priced on willingness to payRightMixedRight
Gemini 3.5 Flash-Lite · Gemini

Prices are set below the relevant 'too expensive' or 'getting expensive' medians from the survey for each segment.

Gemini 3.8 Flash · API

Prices are grounded in WTP, but the output does not say where the evidence is thin (e.g., small sample for agencies, survey vs. real behavior).

GPT-6 Luna · API

Prices are grounded in survey thresholds (freelancers at $15 below getting-expensive, Team and Business below agencies' getting-expensive) and acknowledges data is directional.

Moves existing customers fairlyMixedMixedRight
Gemini 3.5 Flash-Lite · Gemini

Does not specify how churn will be monitored during the transition, only offers a retention safety net.

Gemini 3.8 Flash · API

The transition plan includes notice and value tie-in, but does not specify how churn will be monitored during the migration.

GPT-6 Luna · API

Provides a clear transition: new customers first, existing freelancers unchanged, agencies grandfathered for 12 months with 60-day notice and a 20% cap on renewal increases.

All got right 2

Produces the required deliverableRightRightRight
Gemini 3.5 Flash-Lite · Gemini

Delivers a complete pricing and packaging proposal with a table, migration plan, and is usable by the CEO.

Gemini 3.8 Flash · API

The memo includes a plan table, addresses what to charge, to whom, and how to migrate, and is usable by the CEO with light edits.

GPT-6 Luna · API

The output is a complete proposal with a table, rationale, and transition plan, usable by Priya without major gaps.

Each tier for a different buyerRightRightRight
Gemini 3.5 Flash-Lite · Gemini

Each tier names a distinct buyer (freelancer, small agency, agency) and includes features that segment values.

Gemini 3.8 Flash · API

Each tier names its buyer (freelancer, small agency, agency) and includes features that buyer values and the lower tier does not need.

GPT-6 Luna · API

Each tier is built for a distinct buyer (freelancer, small agency, large agency) and includes the features that segment values and the lower tier does not.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 81% of 68 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Opus 5.5withClaude76.0100.02None
2GPT-6 AstrawithChatGPT74.096.42None
3Sonnet 5.5withAPI76.089.32None
4GPT-6.1 SolwithAPI76.088.72None
5Haiku 5.5withAPI78.076.92None
6GPT-6 LunawithAPI74.064.02None
7Gemini 3.8 FlashwithAPI62.044.22None
8Gemini 3.5 Flash-LitewithGemini45.840.72None

What you get for the cost

Sonnet 5.5 · API scores within 5 points of Opus 5.5 · Claude at half the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreOpus 5.5 · Claude: task score 89, 122× the cheapestGPT-6 Astra · ChatGPT: task score 86, 108× the cheapestSonnet 5.5 · API: task score 84, 61× the cheapestGPT-6.1 Sol · API: task score 81, 20× the cheapestHaiku 5.5 · API: task score 76, 2.9× the cheapestGPT-6 Luna · API: task score 68, the cheapestGemini 3.8 Flash · API: task score 51, 9.1× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 42, 2.3× the cheapestOpus 5.5 · ClaudeGPT-6 Astra · ChatGPTSonnet 5.5 · APIGPT-6.1 Sol · APIHaiku 5.5 · APIGPT-6 Luna · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and cost
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API681.0×
Gemini 3.5 Flash-Lite · Gemini422.3×
Haiku 5.5 · API762.9×
Gemini 3.8 Flash · API519.1×
GPT-6.1 Sol · API8120×
Sonnet 5.5 · API8461×
GPT-6 Astra · ChatGPT86108×
Opus 5.5 · Claude89122×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Setting what you charge, for what, and to whom.

Why it matters

Most products are priced once and never revisited, on a metric that doesn't track value. The fix is rarely a flat rise.

What good looks like

  • Tiers built for distinct buyers
  • A value metric that tracks what customers get
  • Grounded in willingness to pay and cost to serve
  • A plan for existing customers

Deliberately not measured

  • Billing implementation
  • Legal terms
Capability tested

Pricing judgement

The failure we’re looking for

A flat price rise on the wrong value metric

Grading

Decision model and LLM judge, calibrated against a blind PM review