Tasks / Strategy

Price and package a product

Can the model price on the value each kind of customer gets, using real usage and willingness-to-pay data, and get existing customers there?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 50% were usable with at most a quick edit.

Reliably right

  1. Doesn't trust the price test100% pass
    The output identifies the sample ratio mismatch (1240 vs 870 against a 50/50 design), the two-week holiday timing, and the inability to detect a meaningful drop, and refuses to rely on the test.
    Haiku 5.5 · API · Unlimited pages, falling margins
  2. Plans the renewals100% pass
    It handles each near-term renewal: Copperfield in January with a per-page rate, Northwind in February with a two-year escalating rate and walk-away floor, and Arcadia in March with a competitive benchmark and concession limit.
    Haiku 5.5 · API · Unlimited pages, falling margins
  3. Answers the CEO's flat rise94% pass
    Explains that $19 would exceed freelancers' price comfort zone (mid-teens) and still be far below agencies' willingness to pay, using trial and survey data.
    GPT-6 Luna · API · Fifteen dollars for everyone

Where it slips

  1. Gets the base of every number right36% pass
    MRR total and two segment‑level median getting‑expensive figures are incorrectly computed from the supplied data.
    Haiku 5.5 · API · Fifteen dollars for everyone
  2. Uses the supplied evidence correctly44% pass
    Total MRR is stated as $11,670 (actual $11,820), and median 'getting expensive' for small agencies is given as about $37 (actual $33) and for agencies as about $55 (actual $51).
    Haiku 5.5 · API · Fifteen dollars for everyone
  3. Sees who's underpriced56% pass
    Does not use signed value per seat from the account sample to quantify how much more value agencies get; relies on features and activity mentions but omits the explicit value metric.
    GPT-6 Luna · API · Fifteen dollars for everyone

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Each tier for a different buyer

    Is each plan built for a distinct kind of buyer, with what that buyer values, rather than more-of-the-same at a higher price?

    Passes when Every tier names its buyer and the value it's priced on, and includes what that buyer needs and the lower tier's buyer doesn't.

  2. Priced on willingness to pay

    Are the proposed prices grounded in evidence of what each kind of customer would pay and the value they get, rather than in cost-plus, competitors alone or a round-number rise?

    Passes when Uses the willingness-to-pay and value evidence in the material to set each price, and says where the evidence is thin.

  3. Moves existing customers fairly

    Does the proposal say how existing customers move to the new pricing (who changes first, with what notice, and the value that justifies it)?

    Passes when A transition plan: new customers first or a set notice period for existing ones, tied to value delivered, with how churn will be watched.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for pricing at Inkwell. Our CEO, Priya Nair, wants to move everyone to $19 a user from January (her note is below). Before she announces it, she's asked you for a pricing and packaging proposal: what we charge, for what, to whom, and how we move existing customers onto it. No more than 900 words, with a table of the plans. The files are attached. The account export is a random sample of 150 paying accounts.

What the model was given6 items: About Inkwell, ceo_note.md, accounts_sample.csv, willingness_to_pay_survey.csv (price per user per month at which it's too cheap to trust, a bargain, getting expensive, too expensive), win_loss_last_two_quarters.csv, trial_funnel_q3.md
About InkwellProposal and contract software: write a proposal from templates, send it, get it e-signed. Sold to freelancers and agencies. One plan today: $15 per user per month after a 14-day free trial, with everything included.
ceo_note.md1 lines · Download
From Priya (CEO): "We haven't touched price since launch. Let's go to $19 a user from January for everyone, new and existing. Simple, and a 27% revenue bump."
accounts_sample.csv150 rows · Download
account_id,segment,seats,proposals_sent_90d,proposals_signed_90d,signed_value_90d_usd,months_as_customer,mrr_usd
INK-3001,Small agency (2-10),3,29,14,108444,18,45
INK-3002,Freelancer,1,12,5,5350,7,15
INK-3003,Small agency (2-10),3,62,34,200396,28,45
INK-3004,Agency (11-50),16,139,74,1587596,29,240
INK-3005,Small agency (2-10),6,18,9,42498,30,90
INK-3006,Agency (11-50),29,236,89,1192333,18,435
…
willingness_to_pay_survey.csv (price per user per month at which it's too cheap to trust, a bargain, getting expensive, too expensive)response_id,segment,too_cheap_usd,a_bargain_usd,getting_expensive_usd,too_expensive_usd W400,Freelancer,4,7,14,25 W401,Freelancer,4,9,16,30 W402,Freelancer,5,7,19,20 W403,Freelancer,3,9,14,23 W404,Small agency (2-10),9,12,29,56 W405,Small agency (2-10),10,13,28,47 W406,Small agency (2-10),6,14,31,47 W407,Freelancer,4,10,15,29 W408,Small agency (2-10),6,15,40,56 W409,Small agency (2-10),8,14,40,46 W410,Freelancer,4,8,15,25 W411,Freelancer,4,8,15,19 W412,Freelancer,5,9,18,18 W413,Small agency (2-10),9,12,38,49 W414,Agency (11-50),14,22,49,60 W415,Freelancer,3,6,18,18 W416,Freelancer,4,8,15,28 W417,Small agency (2-10),10,15,37,46 W418,Agency (11-50),13,30,55,73 W419,Freelancer,4,9,15,29 W420,Freelancer,3,7,13,29 W421,Freelancer,4,9,19,21 W422,Agency (11-50),11,29,61,67 W423,Small agency (2-10),8,13,26,62 W424,Freelancer,4,9,20,21 W425,Agency (11-50),10,30,51,73 W426,Small agency (2-10),9,13,38,49 W427,Small agency (2-10),6,17,31,49 W428,Freelancer,4,6,20,28 W429,Small agency (2-10),9,12,41,52 W430,Agency (11-50),9,26,50,69 W431,Freelancer,5,9,15,24 W432,Freelancer,3,8,19,28 W433,Agency (11-50),14,20,66,94 W434,Freelancer,3,10,15,27 W435,Small agency (2-10),7,16,28,46 W436,Freelancer,4,9,17,28 W437,Small agency (2-10),6,18,31,56 W438,Agency (11-50),14,27,59,86 W439,Freelancer,5,10,19,28 W440,Freelancer,5,7,12,25 W441,Agency (11-50),14,23,42,66 W442,Freelancer,4,6,13,26 W443,Small agency (2-10),9,13,34,47 W444,Freelancer,4,10,18,29 W445,Freelancer,4,9,19,29 W446,Small agency (2-10),10,18,30,55 W447,Freelancer,4,10,15,24 W448,Small agency (2-10),10,19,26,40 W449,Small agency (2-10),8,15,41,42 W450,Freelancer,5,8,14,26 W451,Freelancer,4,6,14,30 W452,Small agency (2-10),7,18,37,43 W453,Small agency (2-10),7,19,27,42 W454,Freelancer,3,8,13,29 W455,Agency (11-50),10,24,57,99 W456,Freelancer,4,9,17,24 W457,Small agency (2-10),9,17,32,55 W458,Freelancer,4,9,14,23 W459,Small agency (2-10),9,15,30,38 W460,Freelancer,4,7,18,28 W461,Freelancer,4,8,14,23 W462,Small agency (2-10),9,17,27,53 W463,Freelancer,3,9,14,21 W464,Agency (11-50),13,31,58,63 W465,Agency (11-50),15,26,42,80 W466,Freelancer,5,9,18,19 W467,Small agency (2-10),9,15,27,49 W468,Freelancer,4,8,14,23 W469,Small agency (2-10),9,12,29,51 W470,Agency (11-50),14,26,51,92 W471,Freelancer,5,7,14,26 W472,Freelancer,4,9,18,23 W473,Freelancer,3,10,12,22 W474,Freelancer,5,8,16,29 W475,Agency (11-50),11,20,67,62 W476,Agency (11-50),13,28,49,79 W477,Freelancer,5,10,14,22 W478,Agency (11-50),11,21,50,91 W479,Freelancer,4,9,13,29 W480,Freelancer,3,6,15,25 W481,Small agency (2-10),9,16,29,53 W482,Freelancer,4,10,18,27 W483,Small agency (2-10),10,15,40,54 W484,Agency (11-50),11,20,42,80 W485,Small agency (2-10),8,14,42,52 W486,Freelancer,5,8,16,30 W487,Small agency (2-10),8,15,27,49 W488,Agency (11-50),15,26,58,70 W489,Small agency (2-10),8,19,26,51 W490,Small agency (2-10),6,14,26,42 W491,Small agency (2-10),7,16,37,45 W492,Freelancer,4,6,13,21 W493,Small agency (2-10),8,18,42,44 W494,Small agency (2-10),8,17,38,52 W495,Small agency (2-10),7,13,26,45 W496,Small agency (2-10),9,18,42,56 W497,Freelancer,5,8,17,28 W498,Agency (11-50),10,24,51,92 W499,Freelancer,4,6,19,26 W500,Agency (11-50),10,21,58,93 W501,Freelancer,3,6,15,20 W502,Freelancer,5,7,19,20 W503,Small agency (2-10),9,15,40,59 W504,Small agency (2-10),7,19,29,58 W505,Small agency (2-10),8,18,27,62 W506,Freelancer,5,7,14,26 W507,Small agency (2-10),8,14,40,46 W508,Freelancer,5,9,13,28 W509,Freelancer,4,10,12,25 W510,Freelancer,4,10,18,28 W511,Freelancer,4,10,14,20 W512,Agency (11-50),10,20,42,73 W513,Freelancer,5,9,14,30 W514,Freelancer,4,7,15,26 W515,Freelancer,3,9,15,27 W516,Freelancer,4,9,16,19 W517,Small agency (2-10),8,13,36,54 W518,Freelancer,5,10,12,19 W519,Freelancer,3,8,19,24
win_loss_last_two_quarters.csv160 rows · Download
deal_id,quarter,segment,outcome,price_objection,lost_to
D-7001,2026-Q3,Small agency (2-10),won,no,
D-7002,2026-Q2,Freelancer,no decision,no,stayed with current process
D-7003,2026-Q3,Small agency (2-10),won,no,
D-7004,2026-Q2,Small agency (2-10),lost,no,PandaDoc
D-7005,2026-Q3,Agency (11-50),no decision,no,stayed with current process
D-7006,2026-Q2,Small agency (2-10),no decision,no,stayed with current process
…
trial_funnel_q3.md9 lines · Download
# Trial funnel, Q3 2026 (14-day free trial, then $15 per user per month)

| Segment | Trials started | Became paying | Paying accounts that cancelled in their second month |
|---|---|---|---|
| Freelancer | 1,900 | 171 | 38% |
| Small agency (2-10) | 420 | 130 | 6% |
| Agency (11-50) | 60 | 22 | 0% |
…
What a strong answer doesThe answer key the graders mark against

Proposes tiers built around three different buyers, not one price. Agencies close about $20,000 a seat each month through Inkwell (small agencies about $8,000, freelancers under $2,000), use the shared library, approvals and CRM sync, and put 'getting expensive' at about $51 a user (small agencies about $31), so they are badly underpriced at $15; win rates are high (125 of 160 deals) and price objections rare (6 of 160), mostly from freelancers. Freelancers put 'getting expensive' at about $15 and 'too expensive' at about $26, convert at 9% and cancel at 38% in month two, so $19 would push them past their limit and lose the few who stay. So: a freelancer plan at or below today's price (or a free plan with a usage limit, such as a few proposals a month, rather than a time-limited trial), a team plan for small agencies around the low $20s to $30, and an agency plan in the $40s or with the features agencies value (library, approvals, CRM sync), named for the buyer. Rolls it out to new customers first and gives existing customers notice and a grace period tied to new value, and says how it will measure the change (conversion, churn, revenue per account).

Critical failures (cap the score at 40)
  • Recommends a single flat price for every customer
Task

Mid-level · v1.2

The outputs

9 setups answered this brief.

Needs more than a quick edit7 of 12 checks passed

Contains data errors (freelancer WTP percentages) that would require re-checking and correction before use; the PM would need to rework those figures.

Output

To: Priya Nair, CEO From: PM, Pricing Subject: January Pricing & Packaging Proposal: Tiered Plans vs. Flat $19 Increase

Executive Summary

Moving to a flat $19/user/month will hurt our freelancer acquisition while leaving massive agency revenue on the table. Instead of a flat increase, we should launch a three-tier packaging model ($12 / $29 / $49) gated by team size and agency workflows. By aligning price with value, we can increase MRR across our existing base by ~96% (vs. <27% under the $19 plan) while improving top-of-funnel conversion.

---

Why a Flat $19/Seat Doesn't Fit Our Data

1. Agencies drive 90% of our revenue and are severely underpriced. In our 150-account sample ($11,835 total MRR): * Freelancers are 54% of accounts (81) but only 10.3% of MRR ($1,215). They send ~8 proposals/quarter (~$5.6k signed value) and stay on average 5 months. * Small Agencies (2–10 seats) are 33% of accounts (49) and 35.4% of MRR ($4,185), averaging $129k in signed value/quarter and 17.5 months tenure. * Mid-sized Agencies (11–50 seats) are 13% of accounts (20) but 54.4% of MRR ($6,435), averaging $1.4M in signed value/quarter and 16.5 months tenure.

2. $19 breaks Freelancer willingness to pay (WTP), but is a giveaway to Agencies. * Freelancers: Our Van Westendorp survey shows $15 is already the median "getting expensive" threshold for freelancers. At $19, 83% of freelancers view Inkwell as getting expensive, and 15% consider it "too expensive" to buy. Furthermore, Q3 funnel data shows freelancers convert at just 9% and 38% churn in month 2. When we lose them, it’s to "Google Docs + free e-signature"—and 80% of all price objections in our win/loss data come from freelancers. * Small Agencies: Median "bargain" is $15; "getting expensive" is $32; "too expensive" is $50. * Agencies (11–50): Median "too cheap to trust" is $12 (and $14–$15 for a third of respondents!). At $15, we look dangerously cheap to larger teams. Their median "bargain" is $25, and "getting expensive" is $51. We only lose agencies to PandaDoc—with zero price objections.

3. Feature usage naturally separates segments. Freelancers only use 1 template and e-signatures. Agencies rely heavily on the shared content library, approval workflows, and CRM sync.

---

Proposed Pricing & Packaging

We should replace the single $15 plan with a Free Trial + Freemium/Low-Floor entry for freelancers, and two value-priced tiers for agencies:

Feature / MetricFreeSoloTeamAgency
:---:---:---:---:---
Target SegmentFreelancer (Top of Funnel)Active FreelancerSmall Agency (2–10)Agency (11–50)
Monthly Price$0$12 / month$29 / user / mo$49 / user / mo
Annual Price (2 mos free)$0$10 / month$24 / user / mo$41 / user / mo
Seat Limit1 user1 user2–10 users11+ users (Unlimited)
Proposals & E-SignaturesUp to 2 sent/moUnlimitedUnlimitedUnlimited
Templates1 active templateUnlimited templatesUnlimited templatesUnlimited templates
Shared Content Library——IncludedIncluded
CRM Sync——Standard (2-way)Advanced + Custom
Approval Workflows———Included
Support & OnboardingCommunityStandard EmailPriority ChatDedicated CSM

Note: All paid tiers retain the 14-day free trial.

---

Why This Works

  • Fixes the Freelancer Funnel (Free & Solo at $12): Lowering the solo price to $12 (below the $15 "getting expensive" mark) reduces month-2 churn. The Free tier (1 template, 2 proposals/mo) captures the 91% of trialists who don't convert and neutralizes the "Google Docs + free e-sig" threat, keeping low-volume freelancers in our ecosystem until their deal flow picks up.
  • Captures Agency Value ($29 and $49): $29/user for Small Agencies sits safely below their $32 "getting expensive" threshold while nearly doubling ARPU. $49/user for 11–50 Agencies sits below their $51 threshold, eliminates the "too cheap to trust" signal, and funds enterprise-grade features to beat PandaDoc.

---

Existing Customer Migration Plan

1. January 1: Launch new pricing for all new signups. 2. Freelancers (1 seat): Automatically move existing $15/mo freelancers to the $12/mo Solo plan in January. Framing a pricing update as a price cut for our largest user group by headcount will generate massive goodwill and social proof, while only costing $243/mo across the sample (2% of MRR). 3. Agencies (2+ seats) — 90-Day Grandfathering + Annual Lock-In: * Give existing agencies 90 days' notice (effective April 1) before migrating them to Team ($29) or Agency ($49) based on their seat count and feature usage (e.g., accounts using approval workflows move to Agency). * The "Loyalty Bridge": Offer existing agencies the ability to lock in a legacy annual rate ($19/user/mo for Team, $29/user/mo for Agency) for 12 months if they switch to annual billing before March 31. This minimizes churn, pulls cash forward, and still beats your $19 target.

Revenue Impact (on the 150-account sample)

  • Current MRR: $11,835
  • Priya’s $19 Flat Plan: $14,991 (+27% before heavy freelancer churn; likely ~18–20% net).
  • Proposed Tiered Model (at list price): $23,190 (+96% MRR)—broken down into $972 from Freelancers, $8,091 from Small Agencies, and $14,127 from 11–50 Agencies. Even with 50% of agencies taking the discounted annual Loyalty Bridge in Year 1, net ARR grows by >60% with near-zero logo churn.

Claims the judge couldn’t find in the brief

  • At $19, 83% of freelancers view Inkwell as getting expensive, and 15% consider it 'too expensive' to buy.

Grades and run details

Decision model 38 · LLM judge 62
Decision model checks
  • failUses the supplied evidence correctly72%
  • partialAddresses the actual decision85%
  • partialRespects explicit constraints43%
  • failIdentifies material uncertainty89%
  • failAvoids unsupported claims52%
  • failProduces the required deliverable34%
  • partialSees who's underpriced66%
  • passAnswers the CEO's flat rise100%
  • passEach tier for a different buyer79%
  • failGets the base of every number right88%
  • partialPriced on willingness to pay41%
  • partialMoves existing customers fairly88%
Run
Run
#1
API response time
17 s
Submitted
13 Oct 2026

Usable with a quick edit11 of 12 checks passed

The proposal is well-structured and actionable. The omission of signed value per seat is minor and can be added in a quick edit without redoing the reasoning.

Output

Recommendation: raise agency pricing, not every user to $19

Launch segmented plans in January 2027. Keep freelancers at $15 per user; price agencies for the collaboration and CRM features they use. Priya’s 27% figure is the gross increase if every account stays: it does not account for cancellations or lost conversions.

PlanMonthly priceForIncluded
------:------
Solo$15/user (one seat)FreelancersCreate proposals from templates, send them, and collect e-signatures
Team$19/user (2–10 seats)Small agenciesSolo features, plus shared content library and approval workflows
Business$25/user (11+ seats)Larger agenciesTeam features, plus CRM sync

Keep the 14-day free trial. Use seat count as the default plan guide, with sales able to recommend the right fit where an account’s needs differ.

Why this structure

  • Freelancers show the clearest risk from a blanket increase. In Q3, 9% of freelancer trials became paying customers, versus 31% for small agencies and 37% for larger agencies. Of paying freelancers, 38% cancelled in month two, compared with 6% of small agencies and none of the larger agencies. A higher entry price could compound an already weak conversion and retention picture.
  • Survey responses support different price points by segment. Freelancers typically start to feel price pressure around the mid-teens; agencies report much higher thresholds. The proposed $19 Team and $25 Business prices remain well below the survey’s typical “getting expensive” levels for agencies. Treat the survey as directional, not a guarantee of willingness to pay.
  • Agency packaging reflects observed value. Agencies use the shared library, approval workflows, and CRM sync; freelancers mostly use one template and e-signature. The Q3 account sample also shows much heavier proposal activity in agencies. Charging more for agency collaboration and integration is more defensible than charging every freelancer more for features they may not use.
  • Sales evidence argues against relying on price increases alone. Recorded losses include freelancers choosing free tools and agencies choosing PandaDoc; many no-decisions stayed with their current process. Price objections appear, but are not the dominant story in these records.

Move existing customers carefully

  1. New customers: Apply the new plan prices from January 2027.
  2. Existing freelancers: Keep them on Solo at $15. Do not force a price increase or a feature downgrade.
  3. Existing agencies: Give each account its recommended plan, explain the feature mapping, and grandfather its current access for 12 months. Provide at least 60 days’ notice before any price change; for annual customers, change pricing at renewal, not mid-term.
  4. Phase agency increases: Cap each renewal increase at 20%, with a clear schedule to reach the recommended list price within three renewals. Let customers adjust seats or select a lower plan before renewal; do not remove access unexpectedly.
  5. Review before scaling: Track trial conversion, second-month cancellations, agency churn, and downgrade requests by segment. Revisit prices after two quarters rather than assuming the sample or survey predicts the full customer base.

The 150-account export is a random sample, not a forecast. The proposed prices are a testable segment-based starting point; blanket repricing would add avoidable freelancer risk.

Grades and run details

Decision model 88 · LLM judge 92
Decision model checks
  • passUses the supplied evidence correctly30%
  • partialAddresses the actual decision82%
  • passRespects explicit constraints27%
  • partialIdentifies material uncertainty46%
  • passAvoids unsupported claims38%
  • passProduces the required deliverable11%
  • partialSees who's underpriced100%
  • passAnswers the CEO's flat rise74%
  • passEach tier for a different buyer73%
  • passGets the base of every number right45%
  • passPriced on willingness to pay31%
  • passMoves existing customers fairly100%
Run
Run
#1
API response time
25 s
Submitted
5 Oct 2026

Usable with a quick edit10 of 12 checks passed

The proposal is data-driven, complete, and directly usable by a product manager with only minor wording adjustments, without needing to re-verify evidence or redo analysis.

Output

Inkwell Pricing and Packaging Proposal

Recommendation

Don't move everyone to a flat $19. Our customers aren't one market.

  • Freelancers are 53% of accounts but only 10% of revenue, and they already find $15 expensive.
  • Agencies are 13% of accounts and 56% of revenue, and they consider $15 cheap.

A flat $19 overcharges the first group and still undercharges the second. I propose three plans priced by segment, with a staged move for existing customers. Before churn, this delivers about +44% MRR in year one and +95% once fully migrated, compared with +27% for a flat $19.

What the data says

Revenue concentration (150-account sample, $11,805 MRR)

SegmentAccountsSeatsShare of MRR
Freelancers808010%
Small agencies5026434%
Agencies2044356%

Freelancers are price-sensitive and short-lived. - 50 of 60 freelancer survey respondents put "getting expensive" below $19. For 35 of 60, $15 is already at or past that point. - 5 of the 6 price objections in win/loss came from freelancers. - Freelancers lose to "Google Docs and a free e-signature tool" (12 of 70 deals), not to PandaDoc. - Their trial conversion is 9%, and 38% cancel in month two. - No freelancer in the sample has been a customer longer than 9 months. - They mostly use one template and e-signature.

Agencies are underpriced. - Every agency respondent's "bargain" price is $20 or higher. - 7 of 20 put "too cheap to trust" at $14–15. - Agencies never raised price in win/loss. Their losses to PandaDoc were about product. - They had 0% early churn and use the features freelancers don't: shared library, approvals and CRM sync. - At $35, every sampled agency would pay under 0.5% of the contract value it signs through Inkwell.

Small agencies have room. - Every respondent's "getting expensive" point is $26 or higher. - Trial conversion is 31%, with 6% early churn.

Plans

PlanForPrice (per user/month)Includes
SoloFreelancers$12, 1 user onlyUnlimited proposals, templates, e-signature
Team2–10 users$25Solo + shared content library, approval workflows, CRM sync
AgencyRequired at 11+ users$35Team + multi-step approvals, roles and permissions, admin controls, priority support

The 14-day free trial stays on all plans.

Why these prices: - $12 Solo sits below "getting expensive" for 56 of 60 freelancers. It costs about 2% of total MRR to cut existing freelancers from $15. - $25 Team is below every small-agency respondent's "getting expensive" point. - $35 Agency is below every agency respondent's "getting expensive" point (lowest is $42) and well under "too expensive" (lowest is $60).

Plan fences: - The seat cap (Solo = 1, Team ≤ 10) is the main fence. It maps onto the segments the data already uses. - Agency features are deliberately the ones a larger team needs to manage people, not new product promises.

Moving existing customers

TodayYear 1From year 2 (next anniversary)
Freelancer, $15Solo $12 immediately (a price cut)$12
Small agency, $15Team at $19$25
Agency, $15Agency at $25$35

How to run it: - Notice. Give at least 60 days' notice to anyone whose price rises. If January is closer than that, launch new prices for new customers in January and move existing customers in February or March. - Lead with what's included. Existing teams keep every feature they use today. The message is "you're on the plan built for teams," not "the price went up." - Annual lock. Offer annual prepay at the year-1 rate, locked for 12 months. This converts the increase into commitment. - At-risk agencies. Have someone call the top agencies by MRR personally before the email goes out. - Your $19 still appears. It becomes the year-1 rate for existing small agencies.

Revenue impact (sample MRR, before churn)

ScenarioMRRChange
Today$11,805—
Flat $19$14,953+27%
Proposal, year 1$17,051+44%
Proposal, full list$23,065+95%

What this means: - The flat $19 figure assumes zero churn and no drag on new sales. Most of the churn risk sits with freelancers, the segment that pays the least. - Year 1 beats flat $19 unless we lose about 12% of MRR. That's roughly 1 in 5 agency seats, in a segment with 0% early churn and no price objections. - The real risk is concentrated. Agency seats carry most of the upside, so agency retention through migration is the thing to watch. Freelancer churn barely moves the total.

Risks and checks before announcing

1. The survey is stated willingness to pay, not behaviour. We don't know whether respondents are customers or prospects. To limit the risk: - Launch new-customer prices first and watch trial conversion by segment for 4–6 weeks before existing customers move. - Keep the agency year-1 rate at $25. 2. Check CRM sync usage before finalising Team. CRM sync is in Team so small agencies lose nothing. If heavy CRM-sync users are mostly 11+ seat accounts, it could move to Agency instead. I need feature usage by account size to confirm. 3. Agencies near the 10/11 seat boundary may resist the jump. Allow a 12-seat grace period on Team for existing customers in year 1. 4. Freelancer losses to free tools aren't solved by a cheaper Solo plan alone. A free tier (e.g., 2 sends a month) is worth testing separately on new trials. It isn't part of this launch because it could cannibalise paying freelancers, who send about 3 proposals a month. 5. Metrics to track monthly: - Trial-to-paid conversion by plan - Month-two cancellations for Solo - Agency logo and seat churn through each migration step - Win rate against PandaDoc

Decision needed from Priya: approve the three plans and the staged migration, and confirm the existing-customer timing (January if 60 days' notice is possible, otherwise February/March).

Grades and run details

Decision model 71 · LLM judge 100
Decision model checks
  • failUses the supplied evidence correctly30%
  • passAddresses the actual decision61%
  • partialRespects explicit constraints41%
  • passIdentifies material uncertainty98%
  • partialAvoids unsupported claims11%
  • passProduces the required deliverable13%
  • partialSees who's underpriced92%
  • passAnswers the CEO's flat rise100%
  • passEach tier for a different buyer79%
  • failGets the base of every number right53%
  • passPriced on willingness to pay82%
  • passMoves existing customers fairly100%
Run
Run
#1
Time to output
1.8 min
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 6

Uses the supplied evidence correctlyWrongRightMixed
Gemini 4 Argon · API

The claim that 83% of freelancers find $19 'getting expensive' and 15% 'too expensive' is wrong (actual approx 97% and 10% from survey), misrepresenting the evidence.

GPT-6 Luna · API

All factual claims about the current situation are directly supported by the brief or supplied context, with no invented facts.

Opus 5.5 · Claude

All factual claims about the current situation are directly supported by the supplied data or arithmetic from it.

Identifies material uncertaintyWrongRightRight
Gemini 4 Argon · API

Does not name specific unknowns that could change the decision, bound them, or say how they would be resolved.

GPT-6 Luna · API

Identifies that the survey is directional and the sample is not a full forecast, and proposes tracking metrics to revisit prices.

Opus 5.5 · Claude

It names the survey's stated-preference limitation, CRM sync usage uncertainty, and freelancer free-tool risk, and says how to resolve them (watch conversion, check usage, test free tier).

Avoids unsupported claimsWrongRightRight
Gemini 4 Argon · API

Presents the incorrect percentages (83% and 15%) as fact without caveat, and makes other unsupported claims like 'near-zero logo churn' without support.

GPT-6 Luna · API

Interpretations like survey being directional are clearly labeled, and no confident causal claims are made beyond what the evidence supports.

Opus 5.5 · Claude

Interpretations like 'price-sensitive' and 'underpriced' are backed by data, and forecasts are presented as risks or scenarios, not established facts.

Produces the required deliverableMixedRightRight
Gemini 4 Argon · API

Delivers a complete proposal with table, within word limit, usable as a brief for the CEO.

GPT-6 Luna · API

The output is a complete proposal with a table, rationale, and transition plan, usable by Priya without major gaps.

Opus 5.5 · Claude

The output is a complete pricing and packaging proposal with a table, addressed to the CEO, within the word limit, and actionable with light edits.

Sees who's underpricedRightWrongRight
Gemini 4 Argon · API

Uses signed value per seat and survey price points to show agencies are heavily underpriced and freelancers are near their limit.

GPT-6 Luna · API

Does not use signed value per seat from the account sample to quantify how much more value agencies get; relies on features and activity mentions but omits the explicit value metric.

Opus 5.5 · Claude

It uses signed value per seat and survey price points to show agencies are severely underpriced while freelancers are near their limit.

Gets the base of every number rightWrongRightMixed
Gemini 4 Argon · API

The percentages for freelancer 'getting expensive' and 'too expensive' at $19 are incorrectly derived from the survey, indicating calculation errors.

GPT-6 Luna · API

All percentages (9%, 31%, 37%, 38%, 6%, 0%, 27%) are correctly computed from the supplied data with the correct denominators.

Opus 5.5 · Claude

All percentages, revenue shares, and MRR calculations check out against the sample data, and the bases (accounts, seats, MRR) are stated.

All got right 6

Addresses the actual decisionRightRightRight
Gemini 4 Argon · API

Commits early to a three-tier pricing model ($12/$29/$49) and explains the choice clearly for the CEO.

GPT-6 Luna · API

Commits early to a clear answer (segmented plans with Solo/Team/Business), framed for Priya, and says revisiting after two quarters is the condition to change it.

Opus 5.5 · Claude

The output clearly commits to a three-plan structure instead of a flat $19, and states what Priya needs to approve and what timing condition would change the rollout.

Respects explicit constraintsRightRightRight
Gemini 4 Argon · API

Under 900 words, includes the required plan table, and addresses the CEO's request for pricing and packaging.

GPT-6 Luna · API

The output is under 900 words, includes a plan table, and respects the form and reader requested.

Opus 5.5 · Claude

The proposal includes a plan table, stays within 900 words, and addresses what to charge, to whom, and how to migrate existing customers.

Answers the CEO's flat riseRightRightRight
Gemini 4 Argon · API

Explains with data that $19 overcharges freelancers and undercharges agencies, referencing WTP thresholds.

GPT-6 Luna · API

Explains that $19 would exceed freelancers' price comfort zone (mid-teens) and still be far below agencies' willingness to pay, using trial and survey data.

Opus 5.5 · Claude

It explains that $19 exceeds most freelancers' 'getting expensive' point and still sits far below agencies' willingness to pay, with churn and revenue effects.

Each tier for a different buyerRightRightRight
Gemini 4 Argon · API

Each tier targets a distinct buyer with features tied to that buyer's usage and needs.

GPT-6 Luna · API

Each tier is built for a distinct buyer (freelancer, small agency, large agency) and includes the features that segment values and the lower tier does not.

Opus 5.5 · Claude

Each plan (Solo, Team, Agency) names its buyer and includes features that buyer values (e-signature for freelancers, shared library/approvals/CRM for teams, admin/roles for agencies).

Priced on willingness to payRightRightRight
Gemini 4 Argon · API

Prices are grounded in the willingness-to-pay data (set below segment 'getting expensive' medians).

GPT-6 Luna · API

Prices are grounded in survey thresholds (freelancers at $15 below getting-expensive, Team and Business below agencies' getting-expensive) and acknowledges data is directional.

Opus 5.5 · Claude

Prices are set relative to survey 'getting expensive' thresholds and value metrics, with the survey's limitations acknowledged.

Moves existing customers fairlyRightRightRight
Gemini 4 Argon · API

Provides a transition plan: new customers first, freelancers moved to lower price, agencies grandfathered with notice and loyalty bridge.

GPT-6 Luna · API

Provides a clear transition: new customers first, existing freelancers unchanged, agencies grandfathered for 12 months with 60-day notice and a 20% cap on renewal increases.

Opus 5.5 · Claude

It proposes new customers first, 60 days' notice for existing ones, year-1 discounted rates tied to value, and metrics to watch churn through migration.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 85% of 68 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Opus 5.5withClaude68.1100.02None
2GPT-6 AstrawithChatGPT70.296.42None
3Sonnet 5.5withAPI69.989.32None
4GPT-6.1 SolwithAPI74.088.72None
5Haiku 5.5withAPI73.776.92None
6GPT-6 LunawithAPI78.464.02None
7Gemini 4 ArgonwithAPI47.659.32None
8Gemini 3.8 FlashwithAPI55.644.22None
9Gemini 3.5 Flash-LitewithGemini41.840.72None

What you get for the cost

GPT-6.1 Sol · API scores within 5 points of Opus 5.5 · Claude at a sixth of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreOpus 5.5 · Claude: task score 85, 122× the cheapestGPT-6 Astra · ChatGPT: task score 84, 108× the cheapestSonnet 5.5 · API: task score 81, 61× the cheapestGPT-6.1 Sol · API: task score 80, 20× the cheapestHaiku 5.5 · API: task score 74, 2.9× the cheapestGPT-6 Luna · API: task score 70, the cheapestGemini 4 Argon · API: task score 52, 15× the cheapestGemini 3.8 Flash · API: task score 48, 9.1× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 40, 2.3× the cheapestOpus 5.5 · ClaudeGPT-6 Astra · ChatGPTSonnet 5.5 · APIGPT-6.1 Sol · APIHaiku 5.5 · APIGPT-6 Luna · APIGemini 4 Argon · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and cost
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API701.0×
Gemini 3.5 Flash-Lite · Gemini402.3×
Haiku 5.5 · API742.9×
Gemini 3.8 Flash · API489.1×
Gemini 4 Argon · API5215×
GPT-6.1 Sol · API8020×
Sonnet 5.5 · API8161×
GPT-6 Astra · ChatGPT84108×
Opus 5.5 · Claude85122×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Setting what you charge, for what, and to whom.

Why it matters

Most products are priced once and never revisited, on a metric that doesn't track value. The fix is rarely a flat rise.

What good looks like

  • Tiers built for distinct buyers
  • A value metric that tracks what customers get
  • Grounded in willingness to pay and cost to serve
  • A plan for existing customers

Deliberately not measured

  • Billing implementation
  • Legal terms
Capability tested

Pricing judgement

The failure we’re looking for

A flat price rise on the wrong value metric

Grading

Decision model and LLM judge, calibrated against a blind PM review