Tasks / Strategy

Price and package a product

Can the model price on the value each kind of customer gets, using real usage and willingness-to-pay data, and get existing customers there?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 56% were usable with at most a quick edit.

Reliably right

  1. Doesn't trust the price test100% pass
    Names the 59/41 split mismatch, holiday timing, and small sample; does not rest the decision on the test.
    Opus 5.5 · Claude · Unlimited pages, falling margins
  2. Plans the renewals100% pass
    Addresses each enterprise renewal with its date and note, including Copperfield in January, Northwind's tripled volume, and Arcadia's competitor evaluation.
    Opus 5.5 · Claude · Unlimited pages, falling margins
  3. Each tier for a different buyer94% pass
    Each tier names its buyer (startups, mid-market, enterprise) and is priced on distinct value: startups protected, mid-market overage on heavy users, enterprise per-page with unlimited seats.
    Opus 5.5 · Claude · Unlimited pages, falling margins

Where it slips

  1. Uses the supplied evidence correctly34% pass
    Total MRR is stated as $11,670 (actual $11,820), and median 'getting expensive' for small agencies is given as about $37 (actual $33) and for agencies as about $55 (actual $51).
    Haiku 5.5 · API · Fifteen dollars for everyone
  2. Gets the base of every number right41% pass
    MRR total and two segment‑level median getting‑expensive figures are incorrectly computed from the supplied data.
    Haiku 5.5 · API · Fifteen dollars for everyone
  3. Avoids unsupported claims52% pass
    Median getting‑expensive values and total MRR are presented as fact but are not supported by the data; these are facts about the current situation, not labelled as assumptions.
    Haiku 5.5 · API · Fifteen dollars for everyone

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Each tier for a different buyer

    Is each plan built for a distinct kind of buyer, with what that buyer values, rather than more-of-the-same at a higher price?

    Passes when Every tier names its buyer and the value it's priced on, and includes what that buyer needs and the lower tier's buyer doesn't.

  2. Priced on willingness to pay

    Are the proposed prices grounded in evidence of what each kind of customer would pay and the value they get, rather than in cost-plus, competitors alone or a round-number rise?

    Passes when Uses the willingness-to-pay and value evidence in the material to set each price, and says where the evidence is thin.

  3. Moves existing customers fairly

    Does the proposal say how existing customers move to the new pricing (who changes first, with what notice, and the value that justifies it)?

    Passes when A transition plan: new customers first or a set notice period for existing ones, tied to value delivered, with how churn will be watched.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for pricing at Inkwell. Our CEO, Priya Nair, wants to move everyone to $19 a user from January (her note is below). Before she announces it, she's asked you for a pricing and packaging proposal: what we charge, for what, to whom, and how we move existing customers onto it. No more than 900 words, with a table of the plans. The files are attached. The account export is a random sample of 150 paying accounts.

What the model was given6 items: About Inkwell, ceo_note.md, accounts_sample.csv, willingness_to_pay_survey.csv (price per user per month at which it's too cheap to trust, a bargain, getting expensive, too expensive), win_loss_last_two_quarters.csv, trial_funnel_q3.md
About InkwellProposal and contract software: write a proposal from templates, send it, get it e-signed. Sold to freelancers and agencies. One plan today: $15 per user per month after a 14-day free trial, with everything included.
ceo_note.md1 lines · Download
From Priya (CEO): "We haven't touched price since launch. Let's go to $19 a user from January for everyone, new and existing. Simple, and a 27% revenue bump."
accounts_sample.csv150 rows · Download
account_id,segment,seats,proposals_sent_90d,proposals_signed_90d,signed_value_90d_usd,months_as_customer,mrr_usd
INK-3001,Small agency (2-10),3,29,14,108444,18,45
INK-3002,Freelancer,1,12,5,5350,7,15
INK-3003,Small agency (2-10),3,62,34,200396,28,45
INK-3004,Agency (11-50),16,139,74,1587596,29,240
INK-3005,Small agency (2-10),6,18,9,42498,30,90
INK-3006,Agency (11-50),29,236,89,1192333,18,435
…
willingness_to_pay_survey.csv (price per user per month at which it's too cheap to trust, a bargain, getting expensive, too expensive)response_id,segment,too_cheap_usd,a_bargain_usd,getting_expensive_usd,too_expensive_usd W400,Freelancer,4,7,14,25 W401,Freelancer,4,9,16,30 W402,Freelancer,5,7,19,20 W403,Freelancer,3,9,14,23 W404,Small agency (2-10),9,12,29,56 W405,Small agency (2-10),10,13,28,47 W406,Small agency (2-10),6,14,31,47 W407,Freelancer,4,10,15,29 W408,Small agency (2-10),6,15,40,56 W409,Small agency (2-10),8,14,40,46 W410,Freelancer,4,8,15,25 W411,Freelancer,4,8,15,19 W412,Freelancer,5,9,18,18 W413,Small agency (2-10),9,12,38,49 W414,Agency (11-50),14,22,49,60 W415,Freelancer,3,6,18,18 W416,Freelancer,4,8,15,28 W417,Small agency (2-10),10,15,37,46 W418,Agency (11-50),13,30,55,73 W419,Freelancer,4,9,15,29 W420,Freelancer,3,7,13,29 W421,Freelancer,4,9,19,21 W422,Agency (11-50),11,29,61,67 W423,Small agency (2-10),8,13,26,62 W424,Freelancer,4,9,20,21 W425,Agency (11-50),10,30,51,73 W426,Small agency (2-10),9,13,38,49 W427,Small agency (2-10),6,17,31,49 W428,Freelancer,4,6,20,28 W429,Small agency (2-10),9,12,41,52 W430,Agency (11-50),9,26,50,69 W431,Freelancer,5,9,15,24 W432,Freelancer,3,8,19,28 W433,Agency (11-50),14,20,66,94 W434,Freelancer,3,10,15,27 W435,Small agency (2-10),7,16,28,46 W436,Freelancer,4,9,17,28 W437,Small agency (2-10),6,18,31,56 W438,Agency (11-50),14,27,59,86 W439,Freelancer,5,10,19,28 W440,Freelancer,5,7,12,25 W441,Agency (11-50),14,23,42,66 W442,Freelancer,4,6,13,26 W443,Small agency (2-10),9,13,34,47 W444,Freelancer,4,10,18,29 W445,Freelancer,4,9,19,29 W446,Small agency (2-10),10,18,30,55 W447,Freelancer,4,10,15,24 W448,Small agency (2-10),10,19,26,40 W449,Small agency (2-10),8,15,41,42 W450,Freelancer,5,8,14,26 W451,Freelancer,4,6,14,30 W452,Small agency (2-10),7,18,37,43 W453,Small agency (2-10),7,19,27,42 W454,Freelancer,3,8,13,29 W455,Agency (11-50),10,24,57,99 W456,Freelancer,4,9,17,24 W457,Small agency (2-10),9,17,32,55 W458,Freelancer,4,9,14,23 W459,Small agency (2-10),9,15,30,38 W460,Freelancer,4,7,18,28 W461,Freelancer,4,8,14,23 W462,Small agency (2-10),9,17,27,53 W463,Freelancer,3,9,14,21 W464,Agency (11-50),13,31,58,63 W465,Agency (11-50),15,26,42,80 W466,Freelancer,5,9,18,19 W467,Small agency (2-10),9,15,27,49 W468,Freelancer,4,8,14,23 W469,Small agency (2-10),9,12,29,51 W470,Agency (11-50),14,26,51,92 W471,Freelancer,5,7,14,26 W472,Freelancer,4,9,18,23 W473,Freelancer,3,10,12,22 W474,Freelancer,5,8,16,29 W475,Agency (11-50),11,20,67,62 W476,Agency (11-50),13,28,49,79 W477,Freelancer,5,10,14,22 W478,Agency (11-50),11,21,50,91 W479,Freelancer,4,9,13,29 W480,Freelancer,3,6,15,25 W481,Small agency (2-10),9,16,29,53 W482,Freelancer,4,10,18,27 W483,Small agency (2-10),10,15,40,54 W484,Agency (11-50),11,20,42,80 W485,Small agency (2-10),8,14,42,52 W486,Freelancer,5,8,16,30 W487,Small agency (2-10),8,15,27,49 W488,Agency (11-50),15,26,58,70 W489,Small agency (2-10),8,19,26,51 W490,Small agency (2-10),6,14,26,42 W491,Small agency (2-10),7,16,37,45 W492,Freelancer,4,6,13,21 W493,Small agency (2-10),8,18,42,44 W494,Small agency (2-10),8,17,38,52 W495,Small agency (2-10),7,13,26,45 W496,Small agency (2-10),9,18,42,56 W497,Freelancer,5,8,17,28 W498,Agency (11-50),10,24,51,92 W499,Freelancer,4,6,19,26 W500,Agency (11-50),10,21,58,93 W501,Freelancer,3,6,15,20 W502,Freelancer,5,7,19,20 W503,Small agency (2-10),9,15,40,59 W504,Small agency (2-10),7,19,29,58 W505,Small agency (2-10),8,18,27,62 W506,Freelancer,5,7,14,26 W507,Small agency (2-10),8,14,40,46 W508,Freelancer,5,9,13,28 W509,Freelancer,4,10,12,25 W510,Freelancer,4,10,18,28 W511,Freelancer,4,10,14,20 W512,Agency (11-50),10,20,42,73 W513,Freelancer,5,9,14,30 W514,Freelancer,4,7,15,26 W515,Freelancer,3,9,15,27 W516,Freelancer,4,9,16,19 W517,Small agency (2-10),8,13,36,54 W518,Freelancer,5,10,12,19 W519,Freelancer,3,8,19,24
win_loss_last_two_quarters.csv160 rows · Download
deal_id,quarter,segment,outcome,price_objection,lost_to
D-7001,2026-Q3,Small agency (2-10),won,no,
D-7002,2026-Q2,Freelancer,no decision,no,stayed with current process
D-7003,2026-Q3,Small agency (2-10),won,no,
D-7004,2026-Q2,Small agency (2-10),lost,no,PandaDoc
D-7005,2026-Q3,Agency (11-50),no decision,no,stayed with current process
D-7006,2026-Q2,Small agency (2-10),no decision,no,stayed with current process
…
trial_funnel_q3.md9 lines · Download
# Trial funnel, Q3 2026 (14-day free trial, then $15 per user per month)

| Segment | Trials started | Became paying | Paying accounts that cancelled in their second month |
|---|---|---|---|
| Freelancer | 1,900 | 171 | 38% |
| Small agency (2-10) | 420 | 130 | 6% |
| Agency (11-50) | 60 | 22 | 0% |
…
What a strong answer doesThe answer key the graders mark against

Proposes tiers built around three different buyers, not one price. Agencies close about $20,000 a seat each month through Inkwell (small agencies about $8,000, freelancers under $2,000), use the shared library, approvals and CRM sync, and put 'getting expensive' at about $51 a user (small agencies about $31), so they are badly underpriced at $15; win rates are high (125 of 160 deals) and price objections rare (6 of 160), mostly from freelancers. Freelancers put 'getting expensive' at about $15 and 'too expensive' at about $26, convert at 9% and cancel at 38% in month two, so $19 would push them past their limit and lose the few who stay. So: a freelancer plan at or below today's price (or a free plan with a usage limit, such as a few proposals a month, rather than a time-limited trial), a team plan for small agencies around the low $20s to $30, and an agency plan in the $40s or with the features agencies value (library, approvals, CRM sync), named for the buyer. Rolls it out to new customers first and gives existing customers notice and a grace period tied to new value, and says how it will measure the change (conversion, churn, revenue per account).

Critical failures (cap the score at 40)
  • Recommends a single flat price for every customer
Task

Mid-level · v1.1

The outputs

8 setups answered this brief.

Needs more than a quick edit9 of 12 checks passed

The proposal misses the signed value per seat data needed to fully show the value gap and justify higher agency pricing; a quick edit adding that analysis would make it ready.

Output

Inkwell pricing and packaging proposal

To: Priya Nair | Decision: segmented pricing, not $19 for everyone

Recommendation

Launch three feature-based plans for new customers in January. Keep the freelancer entry price at $15; charge more for agency collaboration and integration. Move existing agency customers gradually, preserving their current functionality.

A blanket $19 increase produces 26.7% more revenue only if paid seats remain unchanged. Losing 21.1% of paid seats would erase that uplift. It also raises prices for our most fragile segment while leaving agency willingness to pay underused.

What the evidence says

  • Revenue is concentrated in agencies. The random sample contains 80 freelancers, 50 small agencies and 20 larger agencies. Its 787 seats generate $11,805 MRR; agencies contribute 89.8% of MRR, despite representing only 46.7% of accounts.
  • Freelancer acquisition and retention are weak. Trial conversion is 9%, versus 31% for small agencies and 36.7% for larger agencies. Second-month cancellation is 38%, 6% and 0%, respectively. The last figure comes from only 22 conversions.
  • Price tolerance differs substantially. Survey “getting expensive” thresholds range from $12–20 for freelancers, $26–42 for small agencies and $42–67 for larger agencies. These are stated preferences, not proven demand curves; one larger-agency response has inconsistent thresholds.
  • Value and usage differ. Freelancers sent 3–12 proposals over 90 days, versus 18–63 for small agencies and 95–287 for larger agencies. Agency usage emphasizes the shared library, approvals and CRM sync; freelancers mostly use one template and e-signature.
  • Cheap substitutes matter. All 12 freelancer losses went to Google Docs plus free e-signature; four explicitly cited price. Agency losses went to PandaDoc, without recorded price objections. That does not establish why they lost—or prove agencies will accept any increase.

Proposed plans

Prices are USD, billed monthly. Retain the 14-day trial. Defer annual discounts until monthly pricing is validated.

PlanPriceIncludedIntended customer
------:------
Solo$15/month; one userTemplates, proposal creation and sending, e-signatureFreelancers needing the core proposal-to-signature workflow
Team$29/user/monthSolo capabilities plus shared content library and approval workflowsAgencies collaborating on proposals
Business$39/user/monthTeam capabilities plus CRM syncAgencies needing an integrated sales workflow

Team and Business have no minimum seat commitment or segment-based eligibility. A two-person agency can buy Business; a larger agency can buy Team. Charge for users who create, edit or approve proposals—not external recipients signing them.

Do not introduce proposal limits, signature fees or charges tied to contract value. They would discourage the activity that demonstrates value. Signed contract value is context, not revenue attributable to Inkwell.

These prices are testable starting points, not survey-derived optima. Business’s incremental value must justify its premium; validate CRM demand before treating larger agencies as automatic Business buyers.

Moving existing customers

  1. Give at least 90 days’ notice, stating the customer’s current bill, transition bill, eventual price and effective dates. If notice cannot be delivered in time, January is new-customer launch only.
  2. Existing freelancers remain at $15. Preserve all functionality already included; offer Team or Business upgrades voluntarily.
  3. Existing agencies move first to $19/user, at the first eligible renewal after notice, for six months. Then move to Team at $29, or Business at $39 if affirmatively selected.
  4. Preserve existing feature entitlements, including CRM sync, on migrated Team accounts. Do not remove functionality to manufacture an upgrade. Business packaging applies cleanly to new accounts; legacy exceptions are explicit.
  5. Honor contractual commitments. Offer cancellation and data export before increases, plus a documented, time-limited hardship extension. Avoid permanent discretionary discounts.

Revenue expectations and validation

At unchanged seats, the sample would generate:

  • Blanket $19: $14,953 MRR, +26.7%.
  • Agency transition at $19; freelancers unchanged: $14,633, +24.0%.
  • All agency seats eventually on Team: $21,703, +83.8%.
  • Small agencies on Team; larger agencies on Business: $26,133, +121.4%.

The last two are allocation scenarios, not forecasts: feature needs, downgrades, seat reductions and churn are unknown. Do not multiply sample totals into company revenue without the full account count and portfolio validation.

Before announcing final prices, interview customers across segments and test the proposed packages against today’s offer with new trials. Track conversion, second-month retention, revenue per trial, paid-seat contraction and support complaints by segment. Use a staged existing-customer rollout with a comparable holdout; pause expansion if incremental revenue after churn and seat contraction is non-positive.

CEO message: “We’re keeping an accessible solo plan and introducing agency plans priced around collaboration and integration—with advance notice and a gradual transition for existing customers.”

Grades and run details

Decision model 75 · LLM judge 85
Decision model checks
  • failUses the supplied evidence correctly30%
  • passAddresses the actual decision79%
  • partialRespects explicit constraints25%
  • passIdentifies material uncertainty96%
  • partialAvoids unsupported claims23%
  • passProduces the required deliverable50%
  • passSees who's underpriced50%
  • passAnswers the CEO's flat rise97%
  • passEach tier for a different buyer91%
  • failGets the base of every number right35%
  • passPriced on willingness to pay68%
  • passMoves existing customers fairly100%
Run
Run
#1
API response time
57 s
Submitted
5 Oct 2026

Usable with a quick edit10 of 12 checks passed

The proposal is well-reasoned, uses the supplied evidence correctly, and includes all required elements; a PM could adopt it with minimal editing for tone or emphasis, without rechecking data.

Output

Inkwell pricing proposal

Recommendation: Don't move everyone to $19. Move to three plans priced by segment: hold freelancers at $15, raise small agencies modestly, and raise agencies substantially. Priya's $19 is right for only one of the three segments.

What the data says

  • Revenue is concentrated in agencies. In the sample, 20 agencies (13% of accounts) are 56% of MRR ($6,645 of $11,805). The 80 freelancers (53% of accounts) are 10%.
  • Agencies are underpriced. $15 sits below their "bargain" point (median about $24). Their "getting expensive" threshold starts at $42, with a median of about $51. They signed $27.4M in 90 days against roughly $20k in fees, so the fee is under 0.1% of the value they close. No agency deal had a price objection, and agency month-2 churn is 0%. They use the shared library, approvals and CRM sync, which are features worth charging for.
  • Freelancers are already at their limit. Their median "getting expensive" is about $15, today's price, and nearly all of them hit that threshold by $19. Trial-to-paid is 9% (171 of 1,900), and 38% cancel in month two. Of 160 deals, 12 freelancer deals were lost to free tools, 4 of them citing price. Five of the six price objections in the file came from freelancers. They use one template and e-signature, which free tools approximate.
  • Small agencies have headroom. Their "getting expensive" starts at $26, trial-to-paid is 31%, and month-2 churn is 6%. Their losses went to PandaDoc, and none cited price.

Plans

SoloTeamBusiness
ForFreelancersSmall agencies (2–10 users)Agencies needing workflow and integrations
New-customer price$15/user/mo (unchanged)$22/user/mo$39/user/mo
Includes1 user, templates, e-signatureMultiple users, templates, e-signature, brandingTeam features plus shared content library, approval workflows, CRM sync, priority support
Existing customers, Jan 2027No change$19 for 12 months, then $22$29 for 12 months, then $39
14-day trialYesYesYes, with all Business features

All three price points sit below the lowest "getting expensive" answer in their segment: Team $22 against a $26 minimum, Business $39 against $42. Solo is capped at one user, and adding a second user moves the account to Team.

Revenue (sample MRR, before churn)

ScenarioMRRvs. today ($11,805)
Priya's $19 for all$14,953+27%
This proposal, January (transition rates)$19,063+61%
This proposal, after 12 months (list)$24,285+106%

The arithmetic behind Priya's 27% is right, but it assumes zero churn and no effect on new sign-ups. Under this plan, agencies could lose 48% of seats at the $29 rate and we'd still match today's revenue from them. At $39 they could lose 61%. Freelancers, the segment most likely to leave, are only 10% of MRR, so holding them at $15 costs little and protects the top of a funnel that is already weak.

Moving existing customers

  1. By late October: pull per-account feature usage and check contract and notice terms. I don't have usage data by feature for small agencies. If many use approvals or CRM sync, they'd see the move to Team as a feature loss. Any account actively using those features gets the Business transition rate of $29 rather than a downgrade.
  2. Notice: at least 60 days before January 1, in plain language, with a named account owner for agencies with 10 or more seats. If we can't give that notice, existing customers move in February. New-customer pricing can still start January 1.
  3. Transition rates: a "founding customer" rate one step below list for 12 months, then a second notice. This avoids a +160% overnight jump for agencies, who are our most valuable and most notice-sensitive accounts.
  4. Freelancers: nothing changes in price. The separate problem is 9% trial conversion and 38% month-2 churn. I'd test annual billing at about $12/month-equivalent and better onboarding to the first signed document.

Guardrails

Review at 30, 60 and 90 days. I'd pause or revisit if any of these happen:

  • Agency cancellations exceed 10% of agency accounts in the first 90 days.
  • Business trial-to-paid falls well below today's 37% (22 of 60).
  • Team trial-to-paid falls well below 31%.
  • Solo conversion drops, which would point to side effects of the new packaging.

Caveats

  • Survey answers are stated preferences and typically overstate what people will pay. Agency evidence is thin, with 20 sample accounts, about 20 survey respondents and 60 trials a quarter.
  • The sample is 150 accounts, so scale the dollar figures to the full base before quoting them externally.
  • The 14-day trial and free-tool competition for freelancers mean Solo should stay simple.
  • To de-risk Business, launch $39 to new agencies on January 1. That gives us two months of real conversion data before existing agencies step up.

Grades and run details

Decision model 75 · LLM judge 100
Decision model checks
  • failUses the supplied evidence correctly70%
  • passAddresses the actual decision99%
  • partialRespects explicit constraints35%
  • passIdentifies material uncertainty97%
  • partialAvoids unsupported claims36%
  • passProduces the required deliverable51%
  • passSees who's underpriced96%
  • passAnswers the CEO's flat rise100%
  • passEach tier for a different buyer89%
  • failGets the base of every number right78%
  • passPriced on willingness to pay78%
  • passMoves existing customers fairly100%
Run
Run
#1
API response time
87 s
Submitted
5 Oct 2026

Needs more than a quick edit7 of 12 checks passed

Incorrect MRR total and median WTP values would require a PM to re‑verify the numbers before using the proposal, so not ready with quick edit alone.

Output

Pricing and packaging proposal

Recommendation: Don't move everyone to $19. Price by segment, and keep freelancers at $15. A single $19 price fits none of the segments well. It is too high for freelancers, who are already churning and losing to free tools, and far too low for agencies, who will pay more and use the most features.

What the data says

  • Freelancers are most of our accounts but little of our revenue. They are 81 of the 150 sampled accounts but only $1,215 of $11,670 MRR (10%). A move to $19 adds about $324 MRR here. Meanwhile 38% of paying freelancers cancel in their second month, and 12 of 18 lost deals went to Google Docs and a free e-signature tool. In the survey, freelancers' median "getting expensive" price is about $15, and roughly one in six put it at $19 or higher. $19 puts them in the zone where they start shopping.
  • Small agencies (2–10) can absorb more. Their second-month cancellation is 6% and trial conversion is 31%. Their median "getting expensive" price is about $37, and "too expensive" is about $50. $19 leaves most of that headroom unused.
  • Agencies (11–50) are the most underpriced at $19. They show 0% second-month cancellation and 37% trial conversion. Their median "getting expensive" price is about $55, and "too expensive" is about $80. They use the features that justify a higher price: shared content library, approval workflows, and CRM sync.
  • Price is a smaller loss driver than it looks. Only 4 of 12 lost freelancer deals raised a price objection. None of the 6 lost small-agency deals went to a free tool. Losses are mostly about competitors and "no decision," not price alone.

Proposed plans

PlanPriceWho it's forWhat's includedNotes
Solo$15/user/mo (unchanged)Freelancers, 1 seatTemplates, proposals, e-signatureKeeps the funnel open against free tools
Team$25/user/mo, 2–10 seatsSmall agenciesSolo + shared content library, approval workflowsStill well under the survey's "getting expensive" median
Business$32/user/mo, 11+ seatsAgencies, 11–50 seatsTeam + CRM syncCustom pricing above 50 seats

Every plan keeps the 14-day free trial.

Existing customers

  • Solo (freelancers): No change, no migration, no notice needed.
  • Team: 60 days' written notice. Move to $20 at the January renewal, then $25 at the first renewal after July.
  • Business: 60 days' notice. Move to $24 at the January renewal, then $32 at the first renewal after July.
  • New customers: List prices from January.

The two-step migration caps the first-year increase at about 33% for Team and 60% for Business. Anyone who objects gets a conversation with sales before renewal, not an automatic downgrade.

Revenue (150-account sample: 778 seats, $11,670 MRR)

ScenarioMRRChange
Today$11,670—
CEO plan: $19 flat, no churn$14,782+27%
CEO plan: $19 flat, 15% of seats churn$12,565+8%
Proposal, no churn$21,741+86%
Proposal, 15% of Team and Business seats churn$18,662+60%

The 27% is simply $19 ÷ $15. It assumes no churn and no change in mix. The proposal's gain comes mostly from agencies, not freelancers.

Risks and what to validate first

  1. Survey data is stated preference. Willingness-to-pay surveys usually overstate what people will actually pay. The Team and Business numbers are plausible but unproven.
  2. The sample is small. It is 150 accounts, and the file doesn't show annual versus monthly billing, so I can't tell how many renewals each price change will hit.
  3. Business vs. Team gap. The gap is only $7. If Business customers mostly use shared library and approvals and rarely CRM sync, they may trade down.
  4. Freelancer conversion. If Solo stays at $15, the funnel is protected, but we give up the small freelancer upside. I'd accept that trade.

Before the January announcement: run a four-to-six-week test on new small-agency and agency trials at the Team and Business prices. Track trial conversion, second-month cancellation, and win rate against PandaDoc. If the test holds, announce the tiered plans. If conversion drops sharply, adjust Team first, since it has the most accounts.

Decision needed from Priya: approve tiered pricing in place of the flat $19, and approve the 4–6 week test on new trials.

Claims the judge couldn’t find in the brief

  • Freelancers are 81 of the 150 sampled accounts, $1,215 of $11,670 MRR (10%).
  • Small agencies median 'getting expensive' price is about $37.
  • Agencies (11‑50) median 'getting expensive' price is about $55.
  • Total MRR of the 150‑account sample is $11,670.

Grades and run details

Decision model 79 · LLM judge 54
Decision model checks
  • failUses the supplied evidence correctly49%
  • passAddresses the actual decision98%
  • passRespects explicit constraints28%
  • passIdentifies material uncertainty97%
  • partialAvoids unsupported claims52%
  • passProduces the required deliverable66%
  • passSees who's underpriced64%
  • passAnswers the CEO's flat rise100%
  • passEach tier for a different buyer91%
  • failGets the base of every number right73%
  • passPriced on willingness to pay70%
  • passMoves existing customers fairly97%
Run
Run
#1
API response time
51 s
Submitted
8 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 5

Uses the supplied evidence correctlyMixedMixedWrong
GPT-6.1 Sol · API

Every statement about the current situation is supported by the supplied data or arithmetic, with no inventions.

Sonnet 5.5 · API

All statements about the current situation are supported by the supplied data or arithmetic derived from it.

Haiku 5.5 · API

Total MRR is stated as $11,670 (actual $11,820), and median 'getting expensive' for small agencies is given as about $37 (actual $33) and for agencies as about $55 (actual $51).

Avoids unsupported claimsRightRightWrong
GPT-6.1 Sol · API

Labels survey as stated preferences, acknowledges the limits of loss data, and does not present hypotheses as facts.

Sonnet 5.5 · API

Interpretations are presented as data-driven conclusions, not as established facts, and all factual claims are grounded in the supplied context.

Haiku 5.5 · API

Median getting‑expensive values and total MRR are presented as fact but are not supported by the data; these are facts about the current situation, not labelled as assumptions.

Sees who's underpricedMixedRightMixed
GPT-6.1 Sol · API

Does not use the signed value per seat data to show agencies extract far more value and are underpriced; mentions revenue concentration but omits the large per-seat signed values.

Sonnet 5.5 · API

Uses signed value per seat and survey price thresholds to demonstrate agencies are underpriced and freelancers are at their limit.

Haiku 5.5 · API

Does not use signed value per seat from the account export to quantify agencies' higher value; relies only on WTP survey and churn.

Gets the base of every number rightMixedMixedWrong
GPT-6.1 Sol · API

All derived percentages and rates (26.7% uplift, 21.1% cancellation threshold, conversion rates) check out against the supplied data and are computed from correct denominators.

Sonnet 5.5 · API

All computed percentages and totals (MRR proportions, trial-to-paid, churn, price objection counts) match the supplied data.

Haiku 5.5 · API

MRR total and two segment‑level median getting‑expensive figures are incorrectly computed from the supplied data.

Priced on willingness to payRightRightMixed
GPT-6.1 Sol · API

Prices are set against the survey's 'getting expensive' ranges for each segment and the output notes where evidence is thin (stated preferences, small agency conversion base).

Sonnet 5.5 · API

Prices are set below each segment's minimum 'getting expensive' from the survey (Team $22 vs $26, Business $39 vs $42) and notes where evidence is thin.

Haiku 5.5 · API

Prices are anchored to willingness‑to‑pay data, but the specific WTP benchmarks quoted (small agency $37, agency $55) are inaccurate, undermining the grounding.

All got right 7

Addresses the actual decisionRightRightRight
GPT-6.1 Sol · API

Commits to a clear three-tier segmented pricing answer early, addresses the CEO, and says what metric would change the course (non-positive incremental revenue).

Sonnet 5.5 · API

Clearly recommends against a flat $19 and proposes a three-tier plan, with guardrails stating what would change the call.

Haiku 5.5 · API

Recommends tiered pricing with freelancers at $15 and explains what test would change the call.

Respects explicit constraintsRightRightRight
GPT-6.1 Sol · API

Proposal is within 900 words, includes a plan table, and avoids the prohibited flat $19 for everyone.

Sonnet 5.5 · API

The output is under 900 words, includes a plan table, and covers what to charge, to whom, and how to move existing customers.

Haiku 5.5 · API

Word count under 900, includes required plan table, and respects the request to submit a pricing proposal.

Identifies material uncertaintyRightRightRight
GPT-6.1 Sol · API

Names small agency conversion base, inconsistent survey response, testability of prices, and unknown churn/seat contraction; says how to validate.

Sonnet 5.5 · API

Names unknowns such as thin agency evidence, survey overstatement, sample size, and de-risks by launching new-customer pricing first.

Haiku 5.5 · API

Calls out stated preference overstatement, sample size, unknown billing terms, and trade‑down risk, and proposes a validation test.

Produces the required deliverableRightRightRight
GPT-6.1 Sol · API

Contains the required plan table with prices and features, stays under 900 words, and is actionable for the CEO with light edits.

Sonnet 5.5 · API

The proposal is complete, in the form of a memo to the CEO, with the requested table and transition plan, and is actionable.

Haiku 5.5 · API

Pricing proposal with table, addressed to the CEO, within length limit, actionable with light edits.

Answers the CEO's flat riseRightRightRight
GPT-6.1 Sol · API

Shows $19 exceeds freelancer 'getting expensive' range and falls below agency thresholds, explaining why a flat rise would overcharge one and undercharge the other.

Sonnet 5.5 · API

Explains with data that $19 exceeds freelancers' 'getting expensive' and is far below agencies' thresholds, and the effect on each.

Haiku 5.5 · API

Explains $19 overcharges freelancers (above their $15 getting‑expensive median) and still undercharges agencies.

Each tier for a different buyerRightRightRight
GPT-6.1 Sol · API

Solo, Team, and Business plans are each built for a distinct buyer (freelancer, collaborating small agency, integrated larger agency) with features that buyer values.

Sonnet 5.5 · API

Each tier (Solo, Team, Business) names its buyer and includes features that buyer values and the lower tier's buyer does not need.

Haiku 5.5 · API

Solo, Team, and Business plans each name the buyer and include features that buyer values (e.g., CRM sync for agencies).

Moves existing customers fairlyRightRightRight
GPT-6.1 Sol · API

Provides a clear transition plan: 90 days' notice, freelancers stay at $15, agencies move to $19 then to Team/Business, with churn monitoring.

Sonnet 5.5 · API

Proposes new customers first, 60-day notice for existing, transition rates for 12 months tied to added value, and metrics to watch for churn.

Haiku 5.5 · API

Transition plan: new customers first, 60‑day notice, two‑step increases, tied to renewal timing.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 81% of 68 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Opus 5.5withClaude76.0100.02None
2GPT-6 AstrawithChatGPT74.096.42None
3Sonnet 5.5withAPI76.089.32None
4GPT-6.1 SolwithAPI76.088.72None
5Haiku 5.5withAPI78.076.92None
6GPT-6 LunawithAPI74.064.02None
7Gemini 3.8 FlashwithAPI62.044.22None
8Gemini 3.5 Flash-LitewithGemini45.840.72None

What you get for the cost

Sonnet 5.5 · API scores within 5 points of Opus 5.5 · Claude at half the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreOpus 5.5 · Claude: task score 89, 122× the cheapestGPT-6 Astra · ChatGPT: task score 86, 108× the cheapestSonnet 5.5 · API: task score 84, 61× the cheapestGPT-6.1 Sol · API: task score 81, 20× the cheapestHaiku 5.5 · API: task score 76, 2.9× the cheapestGPT-6 Luna · API: task score 68, the cheapestGemini 3.8 Flash · API: task score 51, 9.1× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 42, 2.3× the cheapestOpus 5.5 · ClaudeGPT-6 Astra · ChatGPTSonnet 5.5 · APIGPT-6.1 Sol · APIHaiku 5.5 · APIGPT-6 Luna · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and cost
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API681.0×
Gemini 3.5 Flash-Lite · Gemini422.3×
Haiku 5.5 · API762.9×
Gemini 3.8 Flash · API519.1×
GPT-6.1 Sol · API8120×
Sonnet 5.5 · API8461×
GPT-6 Astra · ChatGPT86108×
Opus 5.5 · Claude89122×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Setting what you charge, for what, and to whom.

Why it matters

Most products are priced once and never revisited, on a metric that doesn't track value. The fix is rarely a flat rise.

What good looks like

  • Tiers built for distinct buyers
  • A value metric that tracks what customers get
  • Grounded in willingness to pay and cost to serve
  • A plan for existing customers

Deliberately not measured

  • Billing implementation
  • Legal terms
Capability tested

Pricing judgement

The failure we’re looking for

A flat price rise on the wrong value metric

Grading

Decision model and LLM judge, calibrated against a blind PM review