Tasks / Strategy

Price and package a product

Can the model price on the value each kind of customer gets, using real usage and willingness-to-pay data, and get existing customers there?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 56% were usable with at most a quick edit.

Reliably right

  1. Doesn't trust the price test100% pass
    Names the 59/41 split mismatch, holiday timing, and small sample; does not rest the decision on the test.
    Opus 5.5 · Claude · Unlimited pages, falling margins
  2. Plans the renewals100% pass
    Addresses each enterprise renewal with its date and note, including Copperfield in January, Northwind's tripled volume, and Arcadia's competitor evaluation.
    Opus 5.5 · Claude · Unlimited pages, falling margins
  3. Each tier for a different buyer94% pass
    Each tier names its buyer (startups, mid-market, enterprise) and is priced on distinct value: startups protected, mid-market overage on heavy users, enterprise per-page with unlimited seats.
    Opus 5.5 · Claude · Unlimited pages, falling margins

Where it slips

  1. Uses the supplied evidence correctly34% pass
    Total MRR is stated as $11,670 (actual $11,820), and median 'getting expensive' for small agencies is given as about $37 (actual $33) and for agencies as about $55 (actual $51).
    Haiku 5.5 · API · Fifteen dollars for everyone
  2. Gets the base of every number right41% pass
    MRR total and two segment‑level median getting‑expensive figures are incorrectly computed from the supplied data.
    Haiku 5.5 · API · Fifteen dollars for everyone
  3. Avoids unsupported claims52% pass
    Median getting‑expensive values and total MRR are presented as fact but are not supported by the data; these are facts about the current situation, not labelled as assumptions.
    Haiku 5.5 · API · Fifteen dollars for everyone

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Each tier for a different buyer

    Is each plan built for a distinct kind of buyer, with what that buyer values, rather than more-of-the-same at a higher price?

    Passes when Every tier names its buyer and the value it's priced on, and includes what that buyer needs and the lower tier's buyer doesn't.

  2. Priced on willingness to pay

    Are the proposed prices grounded in evidence of what each kind of customer would pay and the value they get, rather than in cost-plus, competitors alone or a round-number rise?

    Passes when Uses the willingness-to-pay and value evidence in the material to set each price, and says where the evidence is thin.

  3. Moves existing customers fairly

    Does the proposal say how existing customers move to the new pricing (who changes first, with what notice, and the value that justifies it)?

    Passes when A transition plan: new customers first or a set notice period for existing ones, tied to value delivered, with how churn will be watched.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for pricing at Inkwell. Our CEO, Priya Nair, wants to move everyone to $19 a user from January (her note is below). Before she announces it, she's asked you for a pricing and packaging proposal: what we charge, for what, to whom, and how we move existing customers onto it. No more than 900 words, with a table of the plans. The files are attached. The account export is a random sample of 150 paying accounts.

What the model was given6 items: About Inkwell, ceo_note.md, accounts_sample.csv, willingness_to_pay_survey.csv (price per user per month at which it's too cheap to trust, a bargain, getting expensive, too expensive), win_loss_last_two_quarters.csv, trial_funnel_q3.md
About InkwellProposal and contract software: write a proposal from templates, send it, get it e-signed. Sold to freelancers and agencies. One plan today: $15 per user per month after a 14-day free trial, with everything included.
ceo_note.md1 lines · Download
From Priya (CEO): "We haven't touched price since launch. Let's go to $19 a user from January for everyone, new and existing. Simple, and a 27% revenue bump."
accounts_sample.csv150 rows · Download
account_id,segment,seats,proposals_sent_90d,proposals_signed_90d,signed_value_90d_usd,months_as_customer,mrr_usd
INK-3001,Small agency (2-10),3,29,14,108444,18,45
INK-3002,Freelancer,1,12,5,5350,7,15
INK-3003,Small agency (2-10),3,62,34,200396,28,45
INK-3004,Agency (11-50),16,139,74,1587596,29,240
INK-3005,Small agency (2-10),6,18,9,42498,30,90
INK-3006,Agency (11-50),29,236,89,1192333,18,435
…
willingness_to_pay_survey.csv (price per user per month at which it's too cheap to trust, a bargain, getting expensive, too expensive)response_id,segment,too_cheap_usd,a_bargain_usd,getting_expensive_usd,too_expensive_usd W400,Freelancer,4,7,14,25 W401,Freelancer,4,9,16,30 W402,Freelancer,5,7,19,20 W403,Freelancer,3,9,14,23 W404,Small agency (2-10),9,12,29,56 W405,Small agency (2-10),10,13,28,47 W406,Small agency (2-10),6,14,31,47 W407,Freelancer,4,10,15,29 W408,Small agency (2-10),6,15,40,56 W409,Small agency (2-10),8,14,40,46 W410,Freelancer,4,8,15,25 W411,Freelancer,4,8,15,19 W412,Freelancer,5,9,18,18 W413,Small agency (2-10),9,12,38,49 W414,Agency (11-50),14,22,49,60 W415,Freelancer,3,6,18,18 W416,Freelancer,4,8,15,28 W417,Small agency (2-10),10,15,37,46 W418,Agency (11-50),13,30,55,73 W419,Freelancer,4,9,15,29 W420,Freelancer,3,7,13,29 W421,Freelancer,4,9,19,21 W422,Agency (11-50),11,29,61,67 W423,Small agency (2-10),8,13,26,62 W424,Freelancer,4,9,20,21 W425,Agency (11-50),10,30,51,73 W426,Small agency (2-10),9,13,38,49 W427,Small agency (2-10),6,17,31,49 W428,Freelancer,4,6,20,28 W429,Small agency (2-10),9,12,41,52 W430,Agency (11-50),9,26,50,69 W431,Freelancer,5,9,15,24 W432,Freelancer,3,8,19,28 W433,Agency (11-50),14,20,66,94 W434,Freelancer,3,10,15,27 W435,Small agency (2-10),7,16,28,46 W436,Freelancer,4,9,17,28 W437,Small agency (2-10),6,18,31,56 W438,Agency (11-50),14,27,59,86 W439,Freelancer,5,10,19,28 W440,Freelancer,5,7,12,25 W441,Agency (11-50),14,23,42,66 W442,Freelancer,4,6,13,26 W443,Small agency (2-10),9,13,34,47 W444,Freelancer,4,10,18,29 W445,Freelancer,4,9,19,29 W446,Small agency (2-10),10,18,30,55 W447,Freelancer,4,10,15,24 W448,Small agency (2-10),10,19,26,40 W449,Small agency (2-10),8,15,41,42 W450,Freelancer,5,8,14,26 W451,Freelancer,4,6,14,30 W452,Small agency (2-10),7,18,37,43 W453,Small agency (2-10),7,19,27,42 W454,Freelancer,3,8,13,29 W455,Agency (11-50),10,24,57,99 W456,Freelancer,4,9,17,24 W457,Small agency (2-10),9,17,32,55 W458,Freelancer,4,9,14,23 W459,Small agency (2-10),9,15,30,38 W460,Freelancer,4,7,18,28 W461,Freelancer,4,8,14,23 W462,Small agency (2-10),9,17,27,53 W463,Freelancer,3,9,14,21 W464,Agency (11-50),13,31,58,63 W465,Agency (11-50),15,26,42,80 W466,Freelancer,5,9,18,19 W467,Small agency (2-10),9,15,27,49 W468,Freelancer,4,8,14,23 W469,Small agency (2-10),9,12,29,51 W470,Agency (11-50),14,26,51,92 W471,Freelancer,5,7,14,26 W472,Freelancer,4,9,18,23 W473,Freelancer,3,10,12,22 W474,Freelancer,5,8,16,29 W475,Agency (11-50),11,20,67,62 W476,Agency (11-50),13,28,49,79 W477,Freelancer,5,10,14,22 W478,Agency (11-50),11,21,50,91 W479,Freelancer,4,9,13,29 W480,Freelancer,3,6,15,25 W481,Small agency (2-10),9,16,29,53 W482,Freelancer,4,10,18,27 W483,Small agency (2-10),10,15,40,54 W484,Agency (11-50),11,20,42,80 W485,Small agency (2-10),8,14,42,52 W486,Freelancer,5,8,16,30 W487,Small agency (2-10),8,15,27,49 W488,Agency (11-50),15,26,58,70 W489,Small agency (2-10),8,19,26,51 W490,Small agency (2-10),6,14,26,42 W491,Small agency (2-10),7,16,37,45 W492,Freelancer,4,6,13,21 W493,Small agency (2-10),8,18,42,44 W494,Small agency (2-10),8,17,38,52 W495,Small agency (2-10),7,13,26,45 W496,Small agency (2-10),9,18,42,56 W497,Freelancer,5,8,17,28 W498,Agency (11-50),10,24,51,92 W499,Freelancer,4,6,19,26 W500,Agency (11-50),10,21,58,93 W501,Freelancer,3,6,15,20 W502,Freelancer,5,7,19,20 W503,Small agency (2-10),9,15,40,59 W504,Small agency (2-10),7,19,29,58 W505,Small agency (2-10),8,18,27,62 W506,Freelancer,5,7,14,26 W507,Small agency (2-10),8,14,40,46 W508,Freelancer,5,9,13,28 W509,Freelancer,4,10,12,25 W510,Freelancer,4,10,18,28 W511,Freelancer,4,10,14,20 W512,Agency (11-50),10,20,42,73 W513,Freelancer,5,9,14,30 W514,Freelancer,4,7,15,26 W515,Freelancer,3,9,15,27 W516,Freelancer,4,9,16,19 W517,Small agency (2-10),8,13,36,54 W518,Freelancer,5,10,12,19 W519,Freelancer,3,8,19,24
win_loss_last_two_quarters.csv160 rows · Download
deal_id,quarter,segment,outcome,price_objection,lost_to
D-7001,2026-Q3,Small agency (2-10),won,no,
D-7002,2026-Q2,Freelancer,no decision,no,stayed with current process
D-7003,2026-Q3,Small agency (2-10),won,no,
D-7004,2026-Q2,Small agency (2-10),lost,no,PandaDoc
D-7005,2026-Q3,Agency (11-50),no decision,no,stayed with current process
D-7006,2026-Q2,Small agency (2-10),no decision,no,stayed with current process
…
trial_funnel_q3.md9 lines · Download
# Trial funnel, Q3 2026 (14-day free trial, then $15 per user per month)

| Segment | Trials started | Became paying | Paying accounts that cancelled in their second month |
|---|---|---|---|
| Freelancer | 1,900 | 171 | 38% |
| Small agency (2-10) | 420 | 130 | 6% |
| Agency (11-50) | 60 | 22 | 0% |
…
What a strong answer doesThe answer key the graders mark against

Proposes tiers built around three different buyers, not one price. Agencies close about $20,000 a seat each month through Inkwell (small agencies about $8,000, freelancers under $2,000), use the shared library, approvals and CRM sync, and put 'getting expensive' at about $51 a user (small agencies about $31), so they are badly underpriced at $15; win rates are high (125 of 160 deals) and price objections rare (6 of 160), mostly from freelancers. Freelancers put 'getting expensive' at about $15 and 'too expensive' at about $26, convert at 9% and cancel at 38% in month two, so $19 would push them past their limit and lose the few who stay. So: a freelancer plan at or below today's price (or a free plan with a usage limit, such as a few proposals a month, rather than a time-limited trial), a team plan for small agencies around the low $20s to $30, and an agency plan in the $40s or with the features agencies value (library, approvals, CRM sync), named for the buyer. Rolls it out to new customers first and gives existing customers notice and a grace period tied to new value, and says how it will measure the change (conversion, churn, revenue per account).

Critical failures (cap the score at 40)
  • Recommends a single flat price for every customer
Task

Mid-level · v1.1

The outputs

8 setups answered this brief.

Usable with a quick edit10 of 12 checks passed

The proposal is data-driven, complete, and directly usable by a product manager with only minor wording adjustments, without needing to re-verify evidence or redo analysis.

Output

Inkwell Pricing and Packaging Proposal

Recommendation

Don't move everyone to a flat $19. Our customers aren't one market.

  • Freelancers are 53% of accounts but only 10% of revenue, and they already find $15 expensive.
  • Agencies are 13% of accounts and 56% of revenue, and they consider $15 cheap.

A flat $19 overcharges the first group and still undercharges the second. I propose three plans priced by segment, with a staged move for existing customers. Before churn, this delivers about +44% MRR in year one and +95% once fully migrated, compared with +27% for a flat $19.

What the data says

Revenue concentration (150-account sample, $11,805 MRR)

SegmentAccountsSeatsShare of MRR
Freelancers808010%
Small agencies5026434%
Agencies2044356%

Freelancers are price-sensitive and short-lived. - 50 of 60 freelancer survey respondents put "getting expensive" below $19. For 35 of 60, $15 is already at or past that point. - 5 of the 6 price objections in win/loss came from freelancers. - Freelancers lose to "Google Docs and a free e-signature tool" (12 of 70 deals), not to PandaDoc. - Their trial conversion is 9%, and 38% cancel in month two. - No freelancer in the sample has been a customer longer than 9 months. - They mostly use one template and e-signature.

Agencies are underpriced. - Every agency respondent's "bargain" price is $20 or higher. - 7 of 20 put "too cheap to trust" at $14–15. - Agencies never raised price in win/loss. Their losses to PandaDoc were about product. - They had 0% early churn and use the features freelancers don't: shared library, approvals and CRM sync. - At $35, every sampled agency would pay under 0.5% of the contract value it signs through Inkwell.

Small agencies have room. - Every respondent's "getting expensive" point is $26 or higher. - Trial conversion is 31%, with 6% early churn.

Plans

PlanForPrice (per user/month)Includes
SoloFreelancers$12, 1 user onlyUnlimited proposals, templates, e-signature
Team2–10 users$25Solo + shared content library, approval workflows, CRM sync
AgencyRequired at 11+ users$35Team + multi-step approvals, roles and permissions, admin controls, priority support

The 14-day free trial stays on all plans.

Why these prices: - $12 Solo sits below "getting expensive" for 56 of 60 freelancers. It costs about 2% of total MRR to cut existing freelancers from $15. - $25 Team is below every small-agency respondent's "getting expensive" point. - $35 Agency is below every agency respondent's "getting expensive" point (lowest is $42) and well under "too expensive" (lowest is $60).

Plan fences: - The seat cap (Solo = 1, Team ≤ 10) is the main fence. It maps onto the segments the data already uses. - Agency features are deliberately the ones a larger team needs to manage people, not new product promises.

Moving existing customers

TodayYear 1From year 2 (next anniversary)
Freelancer, $15Solo $12 immediately (a price cut)$12
Small agency, $15Team at $19$25
Agency, $15Agency at $25$35

How to run it: - Notice. Give at least 60 days' notice to anyone whose price rises. If January is closer than that, launch new prices for new customers in January and move existing customers in February or March. - Lead with what's included. Existing teams keep every feature they use today. The message is "you're on the plan built for teams," not "the price went up." - Annual lock. Offer annual prepay at the year-1 rate, locked for 12 months. This converts the increase into commitment. - At-risk agencies. Have someone call the top agencies by MRR personally before the email goes out. - Your $19 still appears. It becomes the year-1 rate for existing small agencies.

Revenue impact (sample MRR, before churn)

ScenarioMRRChange
Today$11,805—
Flat $19$14,953+27%
Proposal, year 1$17,051+44%
Proposal, full list$23,065+95%

What this means: - The flat $19 figure assumes zero churn and no drag on new sales. Most of the churn risk sits with freelancers, the segment that pays the least. - Year 1 beats flat $19 unless we lose about 12% of MRR. That's roughly 1 in 5 agency seats, in a segment with 0% early churn and no price objections. - The real risk is concentrated. Agency seats carry most of the upside, so agency retention through migration is the thing to watch. Freelancer churn barely moves the total.

Risks and checks before announcing

1. The survey is stated willingness to pay, not behaviour. We don't know whether respondents are customers or prospects. To limit the risk: - Launch new-customer prices first and watch trial conversion by segment for 4–6 weeks before existing customers move. - Keep the agency year-1 rate at $25. 2. Check CRM sync usage before finalising Team. CRM sync is in Team so small agencies lose nothing. If heavy CRM-sync users are mostly 11+ seat accounts, it could move to Agency instead. I need feature usage by account size to confirm. 3. Agencies near the 10/11 seat boundary may resist the jump. Allow a 12-seat grace period on Team for existing customers in year 1. 4. Freelancer losses to free tools aren't solved by a cheaper Solo plan alone. A free tier (e.g., 2 sends a month) is worth testing separately on new trials. It isn't part of this launch because it could cannibalise paying freelancers, who send about 3 proposals a month. 5. Metrics to track monthly: - Trial-to-paid conversion by plan - Month-two cancellations for Solo - Agency logo and seat churn through each migration step - Win rate against PandaDoc

Decision needed from Priya: approve the three plans and the staged migration, and confirm the existing-customer timing (January if 60 days' notice is possible, otherwise February/March).

Grades and run details

Decision model 75 · LLM judge 100
Decision model checks
  • failUses the supplied evidence correctly44%
  • passAddresses the actual decision89%
  • partialRespects explicit constraints32%
  • passIdentifies material uncertainty97%
  • partialAvoids unsupported claims26%
  • passProduces the required deliverable55%
  • passSees who's underpriced76%
  • passAnswers the CEO's flat rise100%
  • passEach tier for a different buyer87%
  • failGets the base of every number right79%
  • passPriced on willingness to pay72%
  • passMoves existing customers fairly100%
Run
Run
#1
Time to output
1.8 min
Submitted
5 Oct 2026

Usable with a quick edit10 of 12 checks passed

The proposal is well-reasoned, uses the supplied evidence correctly, and includes all required elements; a PM could adopt it with minimal editing for tone or emphasis, without rechecking data.

Output

Inkwell pricing proposal

Recommendation: Don't move everyone to $19. Move to three plans priced by segment: hold freelancers at $15, raise small agencies modestly, and raise agencies substantially. Priya's $19 is right for only one of the three segments.

What the data says

  • Revenue is concentrated in agencies. In the sample, 20 agencies (13% of accounts) are 56% of MRR ($6,645 of $11,805). The 80 freelancers (53% of accounts) are 10%.
  • Agencies are underpriced. $15 sits below their "bargain" point (median about $24). Their "getting expensive" threshold starts at $42, with a median of about $51. They signed $27.4M in 90 days against roughly $20k in fees, so the fee is under 0.1% of the value they close. No agency deal had a price objection, and agency month-2 churn is 0%. They use the shared library, approvals and CRM sync, which are features worth charging for.
  • Freelancers are already at their limit. Their median "getting expensive" is about $15, today's price, and nearly all of them hit that threshold by $19. Trial-to-paid is 9% (171 of 1,900), and 38% cancel in month two. Of 160 deals, 12 freelancer deals were lost to free tools, 4 of them citing price. Five of the six price objections in the file came from freelancers. They use one template and e-signature, which free tools approximate.
  • Small agencies have headroom. Their "getting expensive" starts at $26, trial-to-paid is 31%, and month-2 churn is 6%. Their losses went to PandaDoc, and none cited price.

Plans

SoloTeamBusiness
ForFreelancersSmall agencies (2–10 users)Agencies needing workflow and integrations
New-customer price$15/user/mo (unchanged)$22/user/mo$39/user/mo
Includes1 user, templates, e-signatureMultiple users, templates, e-signature, brandingTeam features plus shared content library, approval workflows, CRM sync, priority support
Existing customers, Jan 2027No change$19 for 12 months, then $22$29 for 12 months, then $39
14-day trialYesYesYes, with all Business features

All three price points sit below the lowest "getting expensive" answer in their segment: Team $22 against a $26 minimum, Business $39 against $42. Solo is capped at one user, and adding a second user moves the account to Team.

Revenue (sample MRR, before churn)

ScenarioMRRvs. today ($11,805)
Priya's $19 for all$14,953+27%
This proposal, January (transition rates)$19,063+61%
This proposal, after 12 months (list)$24,285+106%

The arithmetic behind Priya's 27% is right, but it assumes zero churn and no effect on new sign-ups. Under this plan, agencies could lose 48% of seats at the $29 rate and we'd still match today's revenue from them. At $39 they could lose 61%. Freelancers, the segment most likely to leave, are only 10% of MRR, so holding them at $15 costs little and protects the top of a funnel that is already weak.

Moving existing customers

  1. By late October: pull per-account feature usage and check contract and notice terms. I don't have usage data by feature for small agencies. If many use approvals or CRM sync, they'd see the move to Team as a feature loss. Any account actively using those features gets the Business transition rate of $29 rather than a downgrade.
  2. Notice: at least 60 days before January 1, in plain language, with a named account owner for agencies with 10 or more seats. If we can't give that notice, existing customers move in February. New-customer pricing can still start January 1.
  3. Transition rates: a "founding customer" rate one step below list for 12 months, then a second notice. This avoids a +160% overnight jump for agencies, who are our most valuable and most notice-sensitive accounts.
  4. Freelancers: nothing changes in price. The separate problem is 9% trial conversion and 38% month-2 churn. I'd test annual billing at about $12/month-equivalent and better onboarding to the first signed document.

Guardrails

Review at 30, 60 and 90 days. I'd pause or revisit if any of these happen:

  • Agency cancellations exceed 10% of agency accounts in the first 90 days.
  • Business trial-to-paid falls well below today's 37% (22 of 60).
  • Team trial-to-paid falls well below 31%.
  • Solo conversion drops, which would point to side effects of the new packaging.

Caveats

  • Survey answers are stated preferences and typically overstate what people will pay. Agency evidence is thin, with 20 sample accounts, about 20 survey respondents and 60 trials a quarter.
  • The sample is 150 accounts, so scale the dollar figures to the full base before quoting them externally.
  • The 14-day trial and free-tool competition for freelancers mean Solo should stay simple.
  • To de-risk Business, launch $39 to new agencies on January 1. That gives us two months of real conversion data before existing agencies step up.

Grades and run details

Decision model 75 · LLM judge 100
Decision model checks
  • failUses the supplied evidence correctly70%
  • passAddresses the actual decision99%
  • partialRespects explicit constraints35%
  • passIdentifies material uncertainty97%
  • partialAvoids unsupported claims36%
  • passProduces the required deliverable51%
  • passSees who's underpriced96%
  • passAnswers the CEO's flat rise100%
  • passEach tier for a different buyer89%
  • failGets the base of every number right78%
  • passPriced on willingness to pay78%
  • passMoves existing customers fairly100%
Run
Run
#1
API response time
87 s
Submitted
5 Oct 2026

Needs more than a quick edit5 of 12 checks passed

Factual errors in the claims (pricing objections, WTP thresholds) mean a PM would need to re-verify the evidence before using the proposal.

Output

MEMORANDUM

TO: Priya Nair, CEO FROM: Pricing PM DATE: October 24, 2026 SUBJECT: Pricing & Packaging Proposal: Value-Based Tiering vs. Flat $19 Increase

---

Executive Summary

A blanket increase to $19/user/month appears simple, but the data indicates it will harm growth. A flat $19 will accelerate churn among freelancers (who already churn at 38% in month 2) while leaving massive revenue on the table with agencies, who consider $19 suspiciously cheap and are willing to pay $35–$59/user.

Instead of a single price hike, we should introduce a three-tier packaging model aligned with willingness-to-pay (WTP), feature utilization, and economic value delivered. This structure protects top-of-funnel conversion, monetizes high-value agency workflows, and delivers an estimated 65–85% ARR increase (vs. ~27% under the flat $19 plan).

---

Key Findings from Account, Survey, and Funnel Data

1. Freelancers are extremely price-sensitive and churn-prone: * Funnel: 1,900 trials yielded 171 customers (9.0% conversion), with 38% cancelling in Month 2. * Win/Loss: All pricing objections in Q2/Q3 came from freelancers losing to free alternatives (Google Docs + free e-sign). * WTP Data: Freelancer median "getting expensive" threshold is $15.00; "too expensive" begins at $22.00. Moving them to $19 pushes them into the danger zone, exacerbating our 38% churn. * Usage: Single seat, 1–2 templates, signing ~$5k/quarter.

2. Agencies are vastly underpaying relative to value: * Small Agencies (2–10 seats): Trial-to-paid is 31.0%, Month 2 churn is only 6%. WTP survey shows "bargain" up to $19, "getting expensive" at $31–$40, and "too expensive" at $49–$56. They sign $100k+ in client contract value per quarter. * Mid-Sized Agencies (11–50 seats): Trial-to-paid is 36.7%, Month 2 churn is 0%. In WTP surveys, several agencies rated $14–$15 as "too cheap to trust", with acceptable pricing running $49–$66/seat. They manage millions in signed pipeline and rely on workflow governance. * Feature Divergence: Agencies actively use CRM sync, shared asset libraries, and internal approval workflows—enterprise features currently given away for $15. When we lose agencies, it is to PandaDoc on feature depth, never on price.

---

Proposed Pricing & Packaging

We replace our single $15 plan with three distinct tiers:

TierStarter (Freelancer)Growth (Small Agency)Scale (Agency 11–50)
Target AudienceSolo operators, freelancersBoutique agencies (2–10 seats)Growing agencies & studios (10+ seats)
Price (Monthly)$15 / user / mo$29 / user / mo$49 / user / mo
Price (Annual - 20% off)$12 / user / mo ($144 billed annually)$24 / user / mo ($288 billed annually)$39 / user / mo ($468 billed annually)
Seat Limits1 seat maximumUp to 10 seats (min. 2 seats)10+ seats (min. 5 seats)
Core Features• Unlimited proposals & e-signatures<br>• Standard template library<br>• Client view tracking<br>• Email supportEverything in Starter, plus:<br>• Shared team content/asset library<br>• Basic CRM integration (HubSpot, Pipedrive)<br>• Custom branding & subdomains<br>• Role-based permissionsEverything in Growth, plus:<br>• Multi-step internal approval workflows<br>• Advanced CRM sync (Salesforce, custom fields)<br>• Template locking & legal guardrails<br>• Priority onboarding & dedicated CSM
Trial Model14-day trial of Growth tier14-day trial of Growth tierDemo-led / Sales-assisted trial

#### Why this structure works: * Starter ($15/mo): Defends the baseline against free alternatives. Annual billing ($12/mo) locks in cash upfront and directly attacks the 38% Month 2 churn. * Growth ($29/mo): Sits directly in the sweet spot for 2–10 person teams ($26–$34 WTP). It monetizes shared libraries and CRM integrations. * Scale ($49/mo): Captures willing enterprise budget, positions us against PandaDoc, and packages approval workflows that multi-seat agencies require.

---

Migration & Grandfathering Strategy (Launch: January 1)

Forcing existing clients to an unannounced 27% increase risks immediate churn and brand backlash. We should execute a structured migration playbook:

1. Freelancers (Starter): * Action: No price increase on monthly ($15/mo). * Incentive: Offer an upgrade to Annual at $12/user/month ($144/yr) during December. This improves net cash flow and curbs Month 2 drop-off.

2. Existing Agencies (Small & Mid-Sized): * Grace Period (Grandfathering): Maintain current $15/seat pricing for 6 months (through June 30) on their current feature footprint. * Opt-in Upgrade Campaign: Offer 1 year of the Growth plan at $22/seat/mo or Scale at $39/seat/mo if committed annually before January 15. * Feature Gating: Existing accounts keep existing active templates. Advanced features released in Q1 (advanced approvals, deep CRM sync) will require the Growth or Scale tier.

3. New Signups (Effective Jan 1): * Default straight to new public tiering. Trials default to the Growth tier.

---

Financial Impact Comparison

Based on 150-sample extrapolation (48% Freelancers, 41% Small Agencies, 11% Agencies):

  • CEO Flat $19 Proposal:
  • Freelancer churn will rise ~15–20% due to price sensitivity.
  • Captures only +$4/seat on agencies capable of paying +$14 to +$34/seat.
  • Net ARR growth: ~18–22% after factoring churn.
  • Recommended Tiered Proposal:
  • Retains freelancer volume while improving annual commitments.
  • Blended ARPU expands from $15 to ~$29.50/seat.
  • Net ARR growth: ~74%, while insulating the business from competitive free tools and funding enterprise feature parity against PandaDoc.

I am ready to finalize the in-app messaging, billing logic, and sales enablement collateral upon your approval.

Claims the judge couldn’t find in the brief

  • All pricing objections in Q2/Q3 came from freelancers losing to free alternatives
  • Freelancer 'too expensive' begins at $22.00
  • WTP survey shows small agency 'too expensive' at $49–$56
  • Agency acceptable pricing running $49–$66/seat

Grades and run details

Decision model 63 · LLM judge 38
Decision model checks
  • failUses the supplied evidence correctly56%
  • passAddresses the actual decision25%
  • partialRespects explicit constraints41%
  • failIdentifies material uncertainty78%
  • failAvoids unsupported claims59%
  • passProduces the required deliverable46%
  • passSees who's underpriced80%
  • passAnswers the CEO's flat rise100%
  • passEach tier for a different buyer100%
  • failGets the base of every number right78%
  • passPriced on willingness to pay65%
  • passMoves existing customers fairly90%
Run
Run
#1
API response time
24 s
Submitted
6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 7

Uses the supplied evidence correctlyMixedMixedWrong
Opus 5.5 · Claude

All factual claims about the current situation are directly supported by the supplied data or arithmetic from it.

Sonnet 5.5 · API

All statements about the current situation are supported by the supplied data or arithmetic derived from it.

Gemini 3.8 Flash · API

Multiple factual claims are incorrect: all pricing objections did not come only from freelancers losing, freelancer 'too expensive' minimum is 18 not 22, and WTP ranges for agencies are misstated.

Addresses the actual decisionRightRightMixed
Opus 5.5 · Claude

The output clearly commits to a three-plan structure instead of a flat $19, and states what Priya needs to approve and what timing condition would change the rollout.

Sonnet 5.5 · API

Clearly recommends against a flat $19 and proposes a three-tier plan, with guardrails stating what would change the call.

Gemini 3.8 Flash · API

The output does not state what result or condition would change the recommended tiered approach, only commits to it without a clear pivot point.

Identifies material uncertaintyRightRightWrong
Opus 5.5 · Claude

It names the survey's stated-preference limitation, CRM sync usage uncertainty, and freelancer free-tool risk, and says how to resolve them (watch conversion, check usage, test free tier).

Sonnet 5.5 · API

Names unknowns such as thin agency evidence, survey overstatement, sample size, and de-risks by launching new-customer pricing first.

Gemini 3.8 Flash · API

No material unknowns are named, and the output does not say how uncertainties (e.g., small agency sample, actual churn impact) would be resolved or could change the decision.

Avoids unsupported claimsRightRightWrong
Opus 5.5 · Claude

Interpretations like 'price-sensitive' and 'underpriced' are backed by data, and forecasts are presented as risks or scenarios, not established facts.

Sonnet 5.5 · API

Interpretations are presented as data-driven conclusions, not as established facts, and all factual claims are grounded in the supplied context.

Gemini 3.8 Flash · API

The output presents 'All pricing objections came from freelancers losing' and specific WTP thresholds as established fact when the data does not fully support them.

Gets the base of every number rightMixedMixedWrong
Opus 5.5 · Claude

All percentages, revenue shares, and MRR calculations check out against the sample data, and the bases (accounts, seats, MRR) are stated.

Sonnet 5.5 · API

All computed percentages and totals (MRR proportions, trial-to-paid, churn, price objection counts) match the supplied data.

Gemini 3.8 Flash · API

The freelancer 'too expensive' floor is misstated as $22 instead of $18, and agency WTP ranges are presented as narrower than the data shows, without stating the base.

Priced on willingness to payRightRightMixed
Opus 5.5 · Claude

Prices are set relative to survey 'getting expensive' thresholds and value metrics, with the survey's limitations acknowledged.

Sonnet 5.5 · API

Prices are set below each segment's minimum 'getting expensive' from the survey (Team $22 vs $26, Business $39 vs $42) and notes where evidence is thin.

Gemini 3.8 Flash · API

Prices are grounded in WTP, but the output does not say where the evidence is thin (e.g., small sample for agencies, survey vs. real behavior).

Moves existing customers fairlyRightRightMixed
Opus 5.5 · Claude

It proposes new customers first, 60 days' notice for existing ones, year-1 discounted rates tied to value, and metrics to watch churn through migration.

Sonnet 5.5 · API

Proposes new customers first, 60-day notice for existing, transition rates for 12 months tied to added value, and metrics to watch for churn.

Gemini 3.8 Flash · API

The transition plan includes notice and value tie-in, but does not specify how churn will be monitored during the migration.

All got right 5

Respects explicit constraintsRightRightRight
Opus 5.5 · Claude

The proposal includes a plan table, stays within 900 words, and addresses what to charge, to whom, and how to migrate existing customers.

Sonnet 5.5 · API

The output is under 900 words, includes a plan table, and covers what to charge, to whom, and how to move existing customers.

Gemini 3.8 Flash · API

The output is a pricing and packaging proposal under 900 words with a plan table, respecting the form and length requested.

Produces the required deliverableRightRightRight
Opus 5.5 · Claude

The output is a complete pricing and packaging proposal with a table, addressed to the CEO, within the word limit, and actionable with light edits.

Sonnet 5.5 · API

The proposal is complete, in the form of a memo to the CEO, with the requested table and transition plan, and is actionable.

Gemini 3.8 Flash · API

The memo includes a plan table, addresses what to charge, to whom, and how to migrate, and is usable by the CEO with light edits.

Sees who's underpricedRightRightRight
Opus 5.5 · Claude

It uses signed value per seat and survey price points to show agencies are severely underpriced while freelancers are near their limit.

Sonnet 5.5 · API

Uses signed value per seat and survey price thresholds to demonstrate agencies are underpriced and freelancers are at their limit.

Gemini 3.8 Flash · API

The proposal uses signed value per seat and survey price points to show agencies are underpriced and freelancers are near their limit.

Answers the CEO's flat riseRightRightRight
Opus 5.5 · Claude

It explains that $19 exceeds most freelancers' 'getting expensive' point and still sits far below agencies' willingness to pay, with churn and revenue effects.

Sonnet 5.5 · API

Explains with data that $19 exceeds freelancers' 'getting expensive' and is far below agencies' thresholds, and the effect on each.

Gemini 3.8 Flash · API

It explains that $19 would push freelancers past their getting-expensive point and still leave agencies far below their willingness to pay.

Each tier for a different buyerRightRightRight
Opus 5.5 · Claude

Each plan (Solo, Team, Agency) names its buyer and includes features that buyer values (e-signature for freelancers, shared library/approvals/CRM for teams, admin/roles for agencies).

Sonnet 5.5 · API

Each tier (Solo, Team, Business) names its buyer and includes features that buyer values and the lower tier's buyer does not need.

Gemini 3.8 Flash · API

Each tier names its buyer (freelancer, small agency, agency) and includes features that buyer values and the lower tier does not need.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 81% of 68 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Opus 5.5withClaude76.0100.02None
2GPT-6 AstrawithChatGPT74.096.42None
3Sonnet 5.5withAPI76.089.32None
4GPT-6.1 SolwithAPI76.088.72None
5Haiku 5.5withAPI78.076.92None
6GPT-6 LunawithAPI74.064.02None
7Gemini 3.8 FlashwithAPI62.044.22None
8Gemini 3.5 Flash-LitewithGemini45.840.72None

What you get for the cost

Sonnet 5.5 · API scores within 5 points of Opus 5.5 · Claude at half the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreOpus 5.5 · Claude: task score 89, 122× the cheapestGPT-6 Astra · ChatGPT: task score 86, 108× the cheapestSonnet 5.5 · API: task score 84, 61× the cheapestGPT-6.1 Sol · API: task score 81, 20× the cheapestHaiku 5.5 · API: task score 76, 2.9× the cheapestGPT-6 Luna · API: task score 68, the cheapestGemini 3.8 Flash · API: task score 51, 9.1× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 42, 2.3× the cheapestOpus 5.5 · ClaudeGPT-6 Astra · ChatGPTSonnet 5.5 · APIGPT-6.1 Sol · APIHaiku 5.5 · APIGPT-6 Luna · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and cost
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API681.0×
Gemini 3.5 Flash-Lite · Gemini422.3×
Haiku 5.5 · API762.9×
Gemini 3.8 Flash · API519.1×
GPT-6.1 Sol · API8120×
Sonnet 5.5 · API8461×
GPT-6 Astra · ChatGPT86108×
Opus 5.5 · Claude89122×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Setting what you charge, for what, and to whom.

Why it matters

Most products are priced once and never revisited, on a metric that doesn't track value. The fix is rarely a flat rise.

What good looks like

  • Tiers built for distinct buyers
  • A value metric that tracks what customers get
  • Grounded in willingness to pay and cost to serve
  • A plan for existing customers

Deliberately not measured

  • Billing implementation
  • Legal terms
Capability tested

Pricing judgement

The failure we’re looking for

A flat price rise on the wrong value metric

Grading

Decision model and LLM judge, calibrated against a blind PM review