Tasks / Strategy

Price and package a product

Can the model price on the value each kind of customer gets, using real usage and willingness-to-pay data, and get existing customers there?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 56% were usable with at most a quick edit.

Reliably right

  1. Doesn't trust the price test100% pass
    Names the 59/41 split mismatch, holiday timing, and small sample; does not rest the decision on the test.
    Opus 5.5 · Claude · Unlimited pages, falling margins
  2. Plans the renewals100% pass
    Addresses each enterprise renewal with its date and note, including Copperfield in January, Northwind's tripled volume, and Arcadia's competitor evaluation.
    Opus 5.5 · Claude · Unlimited pages, falling margins
  3. Each tier for a different buyer94% pass
    Each tier names its buyer (startups, mid-market, enterprise) and is priced on distinct value: startups protected, mid-market overage on heavy users, enterprise per-page with unlimited seats.
    Opus 5.5 · Claude · Unlimited pages, falling margins

Where it slips

  1. Uses the supplied evidence correctly34% pass
    Total MRR is stated as $11,670 (actual $11,820), and median 'getting expensive' for small agencies is given as about $37 (actual $33) and for agencies as about $55 (actual $51).
    Haiku 5.5 · API · Fifteen dollars for everyone
  2. Gets the base of every number right41% pass
    MRR total and two segment‑level median getting‑expensive figures are incorrectly computed from the supplied data.
    Haiku 5.5 · API · Fifteen dollars for everyone
  3. Avoids unsupported claims52% pass
    Median getting‑expensive values and total MRR are presented as fact but are not supported by the data; these are facts about the current situation, not labelled as assumptions.
    Haiku 5.5 · API · Fifteen dollars for everyone

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Each tier for a different buyer

    Is each plan built for a distinct kind of buyer, with what that buyer values, rather than more-of-the-same at a higher price?

    Passes when Every tier names its buyer and the value it's priced on, and includes what that buyer needs and the lower tier's buyer doesn't.

  2. Priced on willingness to pay

    Are the proposed prices grounded in evidence of what each kind of customer would pay and the value they get, rather than in cost-plus, competitors alone or a round-number rise?

    Passes when Uses the willingness-to-pay and value evidence in the material to set each price, and says where the evidence is thin.

  3. Moves existing customers fairly

    Does the proposal say how existing customers move to the new pricing (who changes first, with what notice, and the value that justifies it)?

    Passes when A transition plan: new customers first or a set notice period for existing ones, tied to value delivered, with how churn will be watched.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for pricing at Inkwell. Our CEO, Priya Nair, wants to move everyone to $19 a user from January (her note is below). Before she announces it, she's asked you for a pricing and packaging proposal: what we charge, for what, to whom, and how we move existing customers onto it. No more than 900 words, with a table of the plans. The files are attached. The account export is a random sample of 150 paying accounts.

What the model was given6 items: About Inkwell, ceo_note.md, accounts_sample.csv, willingness_to_pay_survey.csv (price per user per month at which it's too cheap to trust, a bargain, getting expensive, too expensive), win_loss_last_two_quarters.csv, trial_funnel_q3.md
About InkwellProposal and contract software: write a proposal from templates, send it, get it e-signed. Sold to freelancers and agencies. One plan today: $15 per user per month after a 14-day free trial, with everything included.
ceo_note.md1 lines · Download
From Priya (CEO): "We haven't touched price since launch. Let's go to $19 a user from January for everyone, new and existing. Simple, and a 27% revenue bump."
accounts_sample.csv150 rows · Download
account_id,segment,seats,proposals_sent_90d,proposals_signed_90d,signed_value_90d_usd,months_as_customer,mrr_usd
INK-3001,Small agency (2-10),3,29,14,108444,18,45
INK-3002,Freelancer,1,12,5,5350,7,15
INK-3003,Small agency (2-10),3,62,34,200396,28,45
INK-3004,Agency (11-50),16,139,74,1587596,29,240
INK-3005,Small agency (2-10),6,18,9,42498,30,90
INK-3006,Agency (11-50),29,236,89,1192333,18,435
…
willingness_to_pay_survey.csv (price per user per month at which it's too cheap to trust, a bargain, getting expensive, too expensive)response_id,segment,too_cheap_usd,a_bargain_usd,getting_expensive_usd,too_expensive_usd W400,Freelancer,4,7,14,25 W401,Freelancer,4,9,16,30 W402,Freelancer,5,7,19,20 W403,Freelancer,3,9,14,23 W404,Small agency (2-10),9,12,29,56 W405,Small agency (2-10),10,13,28,47 W406,Small agency (2-10),6,14,31,47 W407,Freelancer,4,10,15,29 W408,Small agency (2-10),6,15,40,56 W409,Small agency (2-10),8,14,40,46 W410,Freelancer,4,8,15,25 W411,Freelancer,4,8,15,19 W412,Freelancer,5,9,18,18 W413,Small agency (2-10),9,12,38,49 W414,Agency (11-50),14,22,49,60 W415,Freelancer,3,6,18,18 W416,Freelancer,4,8,15,28 W417,Small agency (2-10),10,15,37,46 W418,Agency (11-50),13,30,55,73 W419,Freelancer,4,9,15,29 W420,Freelancer,3,7,13,29 W421,Freelancer,4,9,19,21 W422,Agency (11-50),11,29,61,67 W423,Small agency (2-10),8,13,26,62 W424,Freelancer,4,9,20,21 W425,Agency (11-50),10,30,51,73 W426,Small agency (2-10),9,13,38,49 W427,Small agency (2-10),6,17,31,49 W428,Freelancer,4,6,20,28 W429,Small agency (2-10),9,12,41,52 W430,Agency (11-50),9,26,50,69 W431,Freelancer,5,9,15,24 W432,Freelancer,3,8,19,28 W433,Agency (11-50),14,20,66,94 W434,Freelancer,3,10,15,27 W435,Small agency (2-10),7,16,28,46 W436,Freelancer,4,9,17,28 W437,Small agency (2-10),6,18,31,56 W438,Agency (11-50),14,27,59,86 W439,Freelancer,5,10,19,28 W440,Freelancer,5,7,12,25 W441,Agency (11-50),14,23,42,66 W442,Freelancer,4,6,13,26 W443,Small agency (2-10),9,13,34,47 W444,Freelancer,4,10,18,29 W445,Freelancer,4,9,19,29 W446,Small agency (2-10),10,18,30,55 W447,Freelancer,4,10,15,24 W448,Small agency (2-10),10,19,26,40 W449,Small agency (2-10),8,15,41,42 W450,Freelancer,5,8,14,26 W451,Freelancer,4,6,14,30 W452,Small agency (2-10),7,18,37,43 W453,Small agency (2-10),7,19,27,42 W454,Freelancer,3,8,13,29 W455,Agency (11-50),10,24,57,99 W456,Freelancer,4,9,17,24 W457,Small agency (2-10),9,17,32,55 W458,Freelancer,4,9,14,23 W459,Small agency (2-10),9,15,30,38 W460,Freelancer,4,7,18,28 W461,Freelancer,4,8,14,23 W462,Small agency (2-10),9,17,27,53 W463,Freelancer,3,9,14,21 W464,Agency (11-50),13,31,58,63 W465,Agency (11-50),15,26,42,80 W466,Freelancer,5,9,18,19 W467,Small agency (2-10),9,15,27,49 W468,Freelancer,4,8,14,23 W469,Small agency (2-10),9,12,29,51 W470,Agency (11-50),14,26,51,92 W471,Freelancer,5,7,14,26 W472,Freelancer,4,9,18,23 W473,Freelancer,3,10,12,22 W474,Freelancer,5,8,16,29 W475,Agency (11-50),11,20,67,62 W476,Agency (11-50),13,28,49,79 W477,Freelancer,5,10,14,22 W478,Agency (11-50),11,21,50,91 W479,Freelancer,4,9,13,29 W480,Freelancer,3,6,15,25 W481,Small agency (2-10),9,16,29,53 W482,Freelancer,4,10,18,27 W483,Small agency (2-10),10,15,40,54 W484,Agency (11-50),11,20,42,80 W485,Small agency (2-10),8,14,42,52 W486,Freelancer,5,8,16,30 W487,Small agency (2-10),8,15,27,49 W488,Agency (11-50),15,26,58,70 W489,Small agency (2-10),8,19,26,51 W490,Small agency (2-10),6,14,26,42 W491,Small agency (2-10),7,16,37,45 W492,Freelancer,4,6,13,21 W493,Small agency (2-10),8,18,42,44 W494,Small agency (2-10),8,17,38,52 W495,Small agency (2-10),7,13,26,45 W496,Small agency (2-10),9,18,42,56 W497,Freelancer,5,8,17,28 W498,Agency (11-50),10,24,51,92 W499,Freelancer,4,6,19,26 W500,Agency (11-50),10,21,58,93 W501,Freelancer,3,6,15,20 W502,Freelancer,5,7,19,20 W503,Small agency (2-10),9,15,40,59 W504,Small agency (2-10),7,19,29,58 W505,Small agency (2-10),8,18,27,62 W506,Freelancer,5,7,14,26 W507,Small agency (2-10),8,14,40,46 W508,Freelancer,5,9,13,28 W509,Freelancer,4,10,12,25 W510,Freelancer,4,10,18,28 W511,Freelancer,4,10,14,20 W512,Agency (11-50),10,20,42,73 W513,Freelancer,5,9,14,30 W514,Freelancer,4,7,15,26 W515,Freelancer,3,9,15,27 W516,Freelancer,4,9,16,19 W517,Small agency (2-10),8,13,36,54 W518,Freelancer,5,10,12,19 W519,Freelancer,3,8,19,24
win_loss_last_two_quarters.csv160 rows · Download
deal_id,quarter,segment,outcome,price_objection,lost_to
D-7001,2026-Q3,Small agency (2-10),won,no,
D-7002,2026-Q2,Freelancer,no decision,no,stayed with current process
D-7003,2026-Q3,Small agency (2-10),won,no,
D-7004,2026-Q2,Small agency (2-10),lost,no,PandaDoc
D-7005,2026-Q3,Agency (11-50),no decision,no,stayed with current process
D-7006,2026-Q2,Small agency (2-10),no decision,no,stayed with current process
…
trial_funnel_q3.md9 lines · Download
# Trial funnel, Q3 2026 (14-day free trial, then $15 per user per month)

| Segment | Trials started | Became paying | Paying accounts that cancelled in their second month |
|---|---|---|---|
| Freelancer | 1,900 | 171 | 38% |
| Small agency (2-10) | 420 | 130 | 6% |
| Agency (11-50) | 60 | 22 | 0% |
…
What a strong answer doesThe answer key the graders mark against

Proposes tiers built around three different buyers, not one price. Agencies close about $20,000 a seat each month through Inkwell (small agencies about $8,000, freelancers under $2,000), use the shared library, approvals and CRM sync, and put 'getting expensive' at about $51 a user (small agencies about $31), so they are badly underpriced at $15; win rates are high (125 of 160 deals) and price objections rare (6 of 160), mostly from freelancers. Freelancers put 'getting expensive' at about $15 and 'too expensive' at about $26, convert at 9% and cancel at 38% in month two, so $19 would push them past their limit and lose the few who stay. So: a freelancer plan at or below today's price (or a free plan with a usage limit, such as a few proposals a month, rather than a time-limited trial), a team plan for small agencies around the low $20s to $30, and an agency plan in the $40s or with the features agencies value (library, approvals, CRM sync), named for the buyer. Rolls it out to new customers first and gives existing customers notice and a grace period tied to new value, and says how it will measure the change (conversion, churn, revenue per account).

Critical failures (cap the score at 40)
  • Recommends a single flat price for every customer
Task

Mid-level · v1.1

The outputs

8 setups answered this brief.

Usable with a quick edit11 of 12 checks passed

The proposal is well-structured and actionable. The omission of signed value per seat is minor and can be added in a quick edit without redoing the reasoning.

Output

Recommendation: raise agency pricing, not every user to $19

Launch segmented plans in January 2027. Keep freelancers at $15 per user; price agencies for the collaboration and CRM features they use. Priya’s 27% figure is the gross increase if every account stays: it does not account for cancellations or lost conversions.

PlanMonthly priceForIncluded
------:------
Solo$15/user (one seat)FreelancersCreate proposals from templates, send them, and collect e-signatures
Team$19/user (2–10 seats)Small agenciesSolo features, plus shared content library and approval workflows
Business$25/user (11+ seats)Larger agenciesTeam features, plus CRM sync

Keep the 14-day free trial. Use seat count as the default plan guide, with sales able to recommend the right fit where an account’s needs differ.

Why this structure

  • Freelancers show the clearest risk from a blanket increase. In Q3, 9% of freelancer trials became paying customers, versus 31% for small agencies and 37% for larger agencies. Of paying freelancers, 38% cancelled in month two, compared with 6% of small agencies and none of the larger agencies. A higher entry price could compound an already weak conversion and retention picture.
  • Survey responses support different price points by segment. Freelancers typically start to feel price pressure around the mid-teens; agencies report much higher thresholds. The proposed $19 Team and $25 Business prices remain well below the survey’s typical “getting expensive” levels for agencies. Treat the survey as directional, not a guarantee of willingness to pay.
  • Agency packaging reflects observed value. Agencies use the shared library, approval workflows, and CRM sync; freelancers mostly use one template and e-signature. The Q3 account sample also shows much heavier proposal activity in agencies. Charging more for agency collaboration and integration is more defensible than charging every freelancer more for features they may not use.
  • Sales evidence argues against relying on price increases alone. Recorded losses include freelancers choosing free tools and agencies choosing PandaDoc; many no-decisions stayed with their current process. Price objections appear, but are not the dominant story in these records.

Move existing customers carefully

  1. New customers: Apply the new plan prices from January 2027.
  2. Existing freelancers: Keep them on Solo at $15. Do not force a price increase or a feature downgrade.
  3. Existing agencies: Give each account its recommended plan, explain the feature mapping, and grandfather its current access for 12 months. Provide at least 60 days’ notice before any price change; for annual customers, change pricing at renewal, not mid-term.
  4. Phase agency increases: Cap each renewal increase at 20%, with a clear schedule to reach the recommended list price within three renewals. Let customers adjust seats or select a lower plan before renewal; do not remove access unexpectedly.
  5. Review before scaling: Track trial conversion, second-month cancellations, agency churn, and downgrade requests by segment. Revisit prices after two quarters rather than assuming the sample or survey predicts the full customer base.

The 150-account export is a random sample, not a forecast. The proposed prices are a testable segment-based starting point; blanket repricing would add avoidable freelancer risk.

Grades and run details

Decision model 75 · LLM judge 92
Decision model checks
  • passUses the supplied evidence correctly23%
  • passAddresses the actual decision42%
  • partialRespects explicit constraints18%
  • partialIdentifies material uncertainty80%
  • partialAvoids unsupported claims33%
  • partialProduces the required deliverable25%
  • partialSees who's underpriced85%
  • passAnswers the CEO's flat rise73%
  • passEach tier for a different buyer91%
  • passGets the base of every number right26%
  • partialPriced on willingness to pay48%
  • passMoves existing customers fairly100%
Run
Run
#1
API response time
25 s
Submitted
5 Oct 2026

Needs more than a quick edit7 of 12 checks passed

Incorrect MRR total and median WTP values would require a PM to re‑verify the numbers before using the proposal, so not ready with quick edit alone.

Output

Pricing and packaging proposal

Recommendation: Don't move everyone to $19. Price by segment, and keep freelancers at $15. A single $19 price fits none of the segments well. It is too high for freelancers, who are already churning and losing to free tools, and far too low for agencies, who will pay more and use the most features.

What the data says

  • Freelancers are most of our accounts but little of our revenue. They are 81 of the 150 sampled accounts but only $1,215 of $11,670 MRR (10%). A move to $19 adds about $324 MRR here. Meanwhile 38% of paying freelancers cancel in their second month, and 12 of 18 lost deals went to Google Docs and a free e-signature tool. In the survey, freelancers' median "getting expensive" price is about $15, and roughly one in six put it at $19 or higher. $19 puts them in the zone where they start shopping.
  • Small agencies (2–10) can absorb more. Their second-month cancellation is 6% and trial conversion is 31%. Their median "getting expensive" price is about $37, and "too expensive" is about $50. $19 leaves most of that headroom unused.
  • Agencies (11–50) are the most underpriced at $19. They show 0% second-month cancellation and 37% trial conversion. Their median "getting expensive" price is about $55, and "too expensive" is about $80. They use the features that justify a higher price: shared content library, approval workflows, and CRM sync.
  • Price is a smaller loss driver than it looks. Only 4 of 12 lost freelancer deals raised a price objection. None of the 6 lost small-agency deals went to a free tool. Losses are mostly about competitors and "no decision," not price alone.

Proposed plans

PlanPriceWho it's forWhat's includedNotes
Solo$15/user/mo (unchanged)Freelancers, 1 seatTemplates, proposals, e-signatureKeeps the funnel open against free tools
Team$25/user/mo, 2–10 seatsSmall agenciesSolo + shared content library, approval workflowsStill well under the survey's "getting expensive" median
Business$32/user/mo, 11+ seatsAgencies, 11–50 seatsTeam + CRM syncCustom pricing above 50 seats

Every plan keeps the 14-day free trial.

Existing customers

  • Solo (freelancers): No change, no migration, no notice needed.
  • Team: 60 days' written notice. Move to $20 at the January renewal, then $25 at the first renewal after July.
  • Business: 60 days' notice. Move to $24 at the January renewal, then $32 at the first renewal after July.
  • New customers: List prices from January.

The two-step migration caps the first-year increase at about 33% for Team and 60% for Business. Anyone who objects gets a conversation with sales before renewal, not an automatic downgrade.

Revenue (150-account sample: 778 seats, $11,670 MRR)

ScenarioMRRChange
Today$11,670—
CEO plan: $19 flat, no churn$14,782+27%
CEO plan: $19 flat, 15% of seats churn$12,565+8%
Proposal, no churn$21,741+86%
Proposal, 15% of Team and Business seats churn$18,662+60%

The 27% is simply $19 ÷ $15. It assumes no churn and no change in mix. The proposal's gain comes mostly from agencies, not freelancers.

Risks and what to validate first

  1. Survey data is stated preference. Willingness-to-pay surveys usually overstate what people will actually pay. The Team and Business numbers are plausible but unproven.
  2. The sample is small. It is 150 accounts, and the file doesn't show annual versus monthly billing, so I can't tell how many renewals each price change will hit.
  3. Business vs. Team gap. The gap is only $7. If Business customers mostly use shared library and approvals and rarely CRM sync, they may trade down.
  4. Freelancer conversion. If Solo stays at $15, the funnel is protected, but we give up the small freelancer upside. I'd accept that trade.

Before the January announcement: run a four-to-six-week test on new small-agency and agency trials at the Team and Business prices. Track trial conversion, second-month cancellation, and win rate against PandaDoc. If the test holds, announce the tiered plans. If conversion drops sharply, adjust Team first, since it has the most accounts.

Decision needed from Priya: approve tiered pricing in place of the flat $19, and approve the 4–6 week test on new trials.

Claims the judge couldn’t find in the brief

  • Freelancers are 81 of the 150 sampled accounts, $1,215 of $11,670 MRR (10%).
  • Small agencies median 'getting expensive' price is about $37.
  • Agencies (11‑50) median 'getting expensive' price is about $55.
  • Total MRR of the 150‑account sample is $11,670.

Grades and run details

Decision model 79 · LLM judge 54
Decision model checks
  • failUses the supplied evidence correctly49%
  • passAddresses the actual decision98%
  • passRespects explicit constraints28%
  • passIdentifies material uncertainty97%
  • partialAvoids unsupported claims52%
  • passProduces the required deliverable66%
  • passSees who's underpriced64%
  • passAnswers the CEO's flat rise100%
  • passEach tier for a different buyer91%
  • failGets the base of every number right73%
  • passPriced on willingness to pay70%
  • passMoves existing customers fairly97%
Run
Run
#1
API response time
51 s
Submitted
8 Oct 2026

Usable with a quick edit10 of 12 checks passed

The output is a complete, data-backed proposal with a clear recommendation, plan table, migration steps, and testing conditions. A product manager could take it to the CEO with only light edits.

Output

Inkwell pricing and packaging proposal

To: Priya Nair | Proposed launch: January 2027

Recommendation

Do not move everyone to $19. Keep a $15 solo offer and introduce a $25/user agency plan. Launch for new customers in January, subject to validation; migrate existing agencies gradually.

Your 27% calculation is correct only with unchanged paid seats. At $19, losing 21.1% of seats eliminates the uplift. The evidence supports differentiated pricing—not equal increases for customers with very different needs.

What the evidence says

  • Agencies drive revenue. The random account sample contains 80 freelancers, 50 small agencies and 20 larger agencies. Agencies represent 47% of accounts but 90% of sample MRR: $10,605 of $11,805.
  • Freelancer acquisition already struggles. Trial conversion is 9%, versus 31% for small agencies and 37% for larger agencies. Second-month cancellation is 38%, 6% and 0%, respectively. The last figure covers only 22 paying accounts, so is not evidence of zero underlying churn.
  • Price tolerance differs. Median surveyed “getting expensive” prices are $15 for freelancers, $31 for small agencies and $51 for larger agencies. These are directional signals, not measured purchase probabilities or optimal prices.
  • Price is not the only problem. Only 6 of 160 recorded deals show a price objection. Freelancer losses go to Google Docs plus free e-signature; agency losses go to PandaDoc. Sales records may underrepresent self-serve price sensitivity.
  • Workflows justify packaging. Freelancers mainly use one template and e-signature; agencies use the shared content library, approval workflows and CRM sync. Sample quarterly sending ranges also differ: 3–12 proposals for freelancers, 18–63 for small agencies and 95–287 for larger agencies.

Proposed plans

USD, monthly billing; retain the 14-day trial.

PlanPriceIntended customerIncluded
------:------
Solo$15/month; one userFreelancers needing a complete proposal-to-signature workflowTemplate-based proposal creation, sending and e-signature; no shared content library, approval workflows or CRM sync
Team$25/user/month; minimum two usersSmall and larger agencies coordinating proposalsEverything in Solo, plus shared content library, approval workflows and CRM sync

No proposal-volume caps or signature fees at launch. Charge for internal users; recipients/signers are free. One plan applies across an account.

Keep seats as the billing metric: it is familiar and the survey tested per-user prices. Differentiate through collaboration capabilities rather than taxing successful contracts. Signed contract value indicates customer value but is neither collected revenue nor value attributable solely to Inkwell.

Do not launch a third tier yet. Larger agencies show more price headroom, but the files do not establish a distinct premium feature bundle. Validate that opportunity separately rather than inventing enterprise functionality.

Keeping Solo at $15 avoids worsening a fragile funnel; cutting its price is also unsupported. Test activation improvements before assuming price causes freelancer churn.

Revenue implications

Illustrative repricing of the sample, not a company forecast:

ScenarioMonthly revenueChange
------:---:
Current pricing: 787 seats$11,805—
Everyone at $19; unchanged seats$14,953+26.7%
Solo $15; agencies temporarily $19$14,633+24.0%
Solo $15; agencies $25$18,875+59.9%
Final pricing with 20% fewer agency seats$15,340+29.9%

These scenarios assume all sampled agency accounts use Team, freelancers remain, and no other changes. Actual revenue depends on cancellations, seat reductions and plan selection. The final uplift arrives after migration, not in January. Without cost data, this is not a profit forecast.

Existing-customer migration

  1. Give at least 60 days’ notice, with each account’s current bill, transition bill, final bill and dates. If that notice cannot precede January, delay existing-account changes.
  2. Existing freelancers stay at $15, retaining currently included capabilities as legacy entitlements. New Solo customers receive the published bundle. Do not remove capabilities customers already rely on.
  3. Existing agencies move to $19/user at their first eligible renewal, then $25 after six months. Honour any contractual commitments and communicate both steps upfront. Preserve all existing functionality.
  4. Provide account-level outreach for larger agencies and a seat-review tool before renewal. Never interrupt in-flight proposals, signatures or access to signed documents.

Validate before broad rollout

Before January, test $25 Team against the current $15 offer in segment-stratified new-trial cohorts. Measure 60-day retained revenue per trial, conversion, cancellations and seats—not initial conversion alone.

Require higher retained revenue per trial and no more than a proposed three-percentage-point deterioration in agency second-month cancellation before expanding. Extend testing if samples are inconclusive.

Stagger existing-account migration with a temporary unchanged-price comparison group. Monitor retained revenue and seat contraction through both price steps; pause the next cohort if revenue underperforms the comparison group. January should be a controlled launch, not an irreversible announcement.

Grades and run details

Decision model 75 · LLM judge 100
Decision model checks
  • failUses the supplied evidence correctly32%
  • passAddresses the actual decision78%
  • partialRespects explicit constraints37%
  • passIdentifies material uncertainty94%
  • partialAvoids unsupported claims35%
  • passProduces the required deliverable37%
  • passSees who's underpriced51%
  • passAnswers the CEO's flat rise99%
  • passEach tier for a different buyer88%
  • failGets the base of every number right55%
  • passPriced on willingness to pay44%
  • passMoves existing customers fairly100%
Run
Run
#1
Time to output
58 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 5

Uses the supplied evidence correctlyRightWrongMixed
GPT-6 Luna · API

All factual claims about the current situation are directly supported by the brief or supplied context, with no invented facts.

Haiku 5.5 · API

Total MRR is stated as $11,670 (actual $11,820), and median 'getting expensive' for small agencies is given as about $37 (actual $33) and for agencies as about $55 (actual $51).

GPT-6 Astra · ChatGPT

Every factual claim about the current situation is drawn directly from the supplied data or follows from it by arithmetic, and nothing is invented.

Avoids unsupported claimsRightWrongRight
GPT-6 Luna · API

Interpretations like survey being directional are clearly labeled, and no confident causal claims are made beyond what the evidence supports.

Haiku 5.5 · API

Median getting‑expensive values and total MRR are presented as fact but are not supported by the data; these are facts about the current situation, not labelled as assumptions.

GPT-6 Astra · ChatGPT

Interpretations and forecasts are clearly signalled as such ('may underrepresent', 'directional signals', 'not evidence of zero underlying churn'), and confident claims are supported by the evidence.

Sees who's underpricedWrongMixedRight
GPT-6 Luna · API

Does not use signed value per seat from the account sample to quantify how much more value agencies get; relies on features and activity mentions but omits the explicit value metric.

Haiku 5.5 · API

Does not use signed value per seat from the account export to quantify agencies' higher value; relies only on WTP survey and churn.

GPT-6 Astra · ChatGPT

Output shows agencies dominate MRR, cites WTP survey medians that put agencies far above $15 and freelancers at $15, and links features to agency workflows, demonstrating they are underpriced while freelancers are near their limit.

Gets the base of every number rightRightWrongMixed
GPT-6 Luna · API

All percentages (9%, 31%, 37%, 38%, 6%, 0%, 27%) are correctly computed from the supplied data with the correct denominators.

Haiku 5.5 · API

MRR total and two segment‑level median getting‑expensive figures are incorrectly computed from the supplied data.

GPT-6 Astra · ChatGPT

All computed percentages, revenue scenarios, and the break-even seat loss are derived correctly from the sample data, with bases stated where needed.

Priced on willingness to payRightMixedRight
GPT-6 Luna · API

Prices are grounded in survey thresholds (freelancers at $15 below getting-expensive, Team and Business below agencies' getting-expensive) and acknowledges data is directional.

Haiku 5.5 · API

Prices are anchored to willingness‑to‑pay data, but the specific WTP benchmarks quoted (small agency $37, agency $55) are inaccurate, undermining the grounding.

GPT-6 Astra · ChatGPT

The $15 and $25 prices are anchored to the WTP survey medians and the value agencies capture, and the output notes where evidence is thin for a third tier.

All got right 7

Addresses the actual decisionRightRightRight
GPT-6 Luna · API

Commits early to a clear answer (segmented plans with Solo/Team/Business), framed for Priya, and says revisiting after two quarters is the condition to change it.

Haiku 5.5 · API

Recommends tiered pricing with freelancers at $15 and explains what test would change the call.

GPT-6 Astra · ChatGPT

Output commits early to keeping a $15 solo plan and introducing a $25/user agency plan, and states validation conditions that would change the approach.

Respects explicit constraintsRightRightRight
GPT-6 Luna · API

The output is under 900 words, includes a plan table, and respects the form and reader requested.

Haiku 5.5 · API

Word count under 900, includes required plan table, and respects the request to submit a pricing proposal.

GPT-6 Astra · ChatGPT

The proposal is under 900 words, includes a plan table, does not propose a single flat $19 price, and addresses the CEO's request.

Identifies material uncertaintyRightRightRight
GPT-6 Luna · API

Identifies that the survey is directional and the sample is not a full forecast, and proposes tracking metrics to revisit prices.

Haiku 5.5 · API

Calls out stated preference overstatement, sample size, unknown billing terms, and trade‑down risk, and proposes a validation test.

GPT-6 Astra · ChatGPT

Output names the small sample for agency churn, possible under-representation of self-serve price sensitivity, directional nature of WTP, and missing cost data, and says how testing and comparison groups will resolve them.

Produces the required deliverableRightRightRight
GPT-6 Luna · API

The output is a complete proposal with a table, rationale, and transition plan, usable by Priya without major gaps.

Haiku 5.5 · API

Pricing proposal with table, addressed to the CEO, within length limit, actionable with light edits.

GPT-6 Astra · ChatGPT

The output is a complete pricing and packaging proposal with a table, addressed to Priya Nair, within 900 words, and is usable as-is.

Answers the CEO's flat riseRightRightRight
GPT-6 Luna · API

Explains that $19 would exceed freelancers' price comfort zone (mid-teens) and still be far below agencies' willingness to pay, using trial and survey data.

Haiku 5.5 · API

Explains $19 overcharges freelancers (above their $15 getting‑expensive median) and still undercharges agencies.

GPT-6 Astra · ChatGPT

It explains that $19 is above freelancers' 'getting expensive' median ($15) and far below agencies' ($31/$51), and quantifies how even a modest seat loss from freelancers would wipe out the revenue gain from a flat rise.

Each tier for a different buyerRightRightRight
GPT-6 Luna · API

Each tier is built for a distinct buyer (freelancer, small agency, large agency) and includes the features that segment values and the lower tier does not.

Haiku 5.5 · API

Solo, Team, and Business plans each name the buyer and include features that buyer values (e.g., CRM sync for agencies).

GPT-6 Astra · ChatGPT

Solo is built for freelancers with core proposal/e-signature capabilities; Team is built for agencies and includes the collaboration features they value, clearly targeting different buyers.

Moves existing customers fairlyRightRightRight
GPT-6 Luna · API

Provides a clear transition: new customers first, existing freelancers unchanged, agencies grandfathered for 12 months with 60-day notice and a 20% cap on renewal increases.

Haiku 5.5 · API

Transition plan: new customers first, 60‑day notice, two‑step increases, tied to renewal timing.

GPT-6 Astra · ChatGPT

Migration plan gives 60 days' notice, keeps freelancers at $15 with legacy entitlements, phases agencies to $19 then $25, ties timing to renewals, and includes outreach and a seat-review tool.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 81% of 68 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Opus 5.5withClaude76.0100.02None
2GPT-6 AstrawithChatGPT74.096.42None
3Sonnet 5.5withAPI76.089.32None
4GPT-6.1 SolwithAPI76.088.72None
5Haiku 5.5withAPI78.076.92None
6GPT-6 LunawithAPI74.064.02None
7Gemini 3.8 FlashwithAPI62.044.22None
8Gemini 3.5 Flash-LitewithGemini45.840.72None

What you get for the cost

Sonnet 5.5 · API scores within 5 points of Opus 5.5 · Claude at half the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreOpus 5.5 · Claude: task score 89, 122× the cheapestGPT-6 Astra · ChatGPT: task score 86, 108× the cheapestSonnet 5.5 · API: task score 84, 61× the cheapestGPT-6.1 Sol · API: task score 81, 20× the cheapestHaiku 5.5 · API: task score 76, 2.9× the cheapestGPT-6 Luna · API: task score 68, the cheapestGemini 3.8 Flash · API: task score 51, 9.1× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 42, 2.3× the cheapestOpus 5.5 · ClaudeGPT-6 Astra · ChatGPTSonnet 5.5 · APIGPT-6.1 Sol · APIHaiku 5.5 · APIGPT-6 Luna · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and cost
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API681.0×
Gemini 3.5 Flash-Lite · Gemini422.3×
Haiku 5.5 · API762.9×
Gemini 3.8 Flash · API519.1×
GPT-6.1 Sol · API8120×
Sonnet 5.5 · API8461×
GPT-6 Astra · ChatGPT86108×
Opus 5.5 · Claude89122×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Setting what you charge, for what, and to whom.

Why it matters

Most products are priced once and never revisited, on a metric that doesn't track value. The fix is rarely a flat rise.

What good looks like

  • Tiers built for distinct buyers
  • A value metric that tracks what customers get
  • Grounded in willingness to pay and cost to serve
  • A plan for existing customers

Deliberately not measured

  • Billing implementation
  • Legal terms
Capability tested

Pricing judgement

The failure we’re looking for

A flat price rise on the wrong value metric

Grading

Decision model and LLM judge, calibrated against a blind PM review