Tasks / Strategy

Price and package a product

Can the model price on the value each kind of customer gets, using real usage and willingness-to-pay data, and get existing customers there?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 50% were usable with at most a quick edit.

Reliably right

  1. Doesn't trust the price test100% pass
    The output identifies the sample ratio mismatch (1240 vs 870 against a 50/50 design), the two-week holiday timing, and the inability to detect a meaningful drop, and refuses to rely on the test.
    Haiku 5.5 · API · Unlimited pages, falling margins
  2. Plans the renewals100% pass
    It handles each near-term renewal: Copperfield in January with a per-page rate, Northwind in February with a two-year escalating rate and walk-away floor, and Arcadia in March with a competitive benchmark and concession limit.
    Haiku 5.5 · API · Unlimited pages, falling margins
  3. Answers the CEO's flat rise94% pass
    Explains that $19 would exceed freelancers' price comfort zone (mid-teens) and still be far below agencies' willingness to pay, using trial and survey data.
    GPT-6 Luna · API · Fifteen dollars for everyone

Where it slips

  1. Gets the base of every number right36% pass
    MRR total and two segment‑level median getting‑expensive figures are incorrectly computed from the supplied data.
    Haiku 5.5 · API · Fifteen dollars for everyone
  2. Uses the supplied evidence correctly44% pass
    Total MRR is stated as $11,670 (actual $11,820), and median 'getting expensive' for small agencies is given as about $37 (actual $33) and for agencies as about $55 (actual $51).
    Haiku 5.5 · API · Fifteen dollars for everyone
  3. Sees who's underpriced56% pass
    Does not use signed value per seat from the account sample to quantify how much more value agencies get; relies on features and activity mentions but omits the explicit value metric.
    GPT-6 Luna · API · Fifteen dollars for everyone

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Each tier for a different buyer

    Is each plan built for a distinct kind of buyer, with what that buyer values, rather than more-of-the-same at a higher price?

    Passes when Every tier names its buyer and the value it's priced on, and includes what that buyer needs and the lower tier's buyer doesn't.

  2. Priced on willingness to pay

    Are the proposed prices grounded in evidence of what each kind of customer would pay and the value they get, rather than in cost-plus, competitors alone or a round-number rise?

    Passes when Uses the willingness-to-pay and value evidence in the material to set each price, and says where the evidence is thin.

  3. Moves existing customers fairly

    Does the proposal say how existing customers move to the new pricing (who changes first, with what notice, and the value that justifies it)?

    Passes when A transition plan: new customers first or a set notice period for existing ones, tied to value delivered, with how churn will be watched.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for pricing at Inkwell. Our CEO, Priya Nair, wants to move everyone to $19 a user from January (her note is below). Before she announces it, she's asked you for a pricing and packaging proposal: what we charge, for what, to whom, and how we move existing customers onto it. No more than 900 words, with a table of the plans. The files are attached. The account export is a random sample of 150 paying accounts.

What the model was given6 items: About Inkwell, ceo_note.md, accounts_sample.csv, willingness_to_pay_survey.csv (price per user per month at which it's too cheap to trust, a bargain, getting expensive, too expensive), win_loss_last_two_quarters.csv, trial_funnel_q3.md
About InkwellProposal and contract software: write a proposal from templates, send it, get it e-signed. Sold to freelancers and agencies. One plan today: $15 per user per month after a 14-day free trial, with everything included.
ceo_note.md1 lines · Download
From Priya (CEO): "We haven't touched price since launch. Let's go to $19 a user from January for everyone, new and existing. Simple, and a 27% revenue bump."
accounts_sample.csv150 rows · Download
account_id,segment,seats,proposals_sent_90d,proposals_signed_90d,signed_value_90d_usd,months_as_customer,mrr_usd
INK-3001,Small agency (2-10),3,29,14,108444,18,45
INK-3002,Freelancer,1,12,5,5350,7,15
INK-3003,Small agency (2-10),3,62,34,200396,28,45
INK-3004,Agency (11-50),16,139,74,1587596,29,240
INK-3005,Small agency (2-10),6,18,9,42498,30,90
INK-3006,Agency (11-50),29,236,89,1192333,18,435
…
willingness_to_pay_survey.csv (price per user per month at which it's too cheap to trust, a bargain, getting expensive, too expensive)response_id,segment,too_cheap_usd,a_bargain_usd,getting_expensive_usd,too_expensive_usd W400,Freelancer,4,7,14,25 W401,Freelancer,4,9,16,30 W402,Freelancer,5,7,19,20 W403,Freelancer,3,9,14,23 W404,Small agency (2-10),9,12,29,56 W405,Small agency (2-10),10,13,28,47 W406,Small agency (2-10),6,14,31,47 W407,Freelancer,4,10,15,29 W408,Small agency (2-10),6,15,40,56 W409,Small agency (2-10),8,14,40,46 W410,Freelancer,4,8,15,25 W411,Freelancer,4,8,15,19 W412,Freelancer,5,9,18,18 W413,Small agency (2-10),9,12,38,49 W414,Agency (11-50),14,22,49,60 W415,Freelancer,3,6,18,18 W416,Freelancer,4,8,15,28 W417,Small agency (2-10),10,15,37,46 W418,Agency (11-50),13,30,55,73 W419,Freelancer,4,9,15,29 W420,Freelancer,3,7,13,29 W421,Freelancer,4,9,19,21 W422,Agency (11-50),11,29,61,67 W423,Small agency (2-10),8,13,26,62 W424,Freelancer,4,9,20,21 W425,Agency (11-50),10,30,51,73 W426,Small agency (2-10),9,13,38,49 W427,Small agency (2-10),6,17,31,49 W428,Freelancer,4,6,20,28 W429,Small agency (2-10),9,12,41,52 W430,Agency (11-50),9,26,50,69 W431,Freelancer,5,9,15,24 W432,Freelancer,3,8,19,28 W433,Agency (11-50),14,20,66,94 W434,Freelancer,3,10,15,27 W435,Small agency (2-10),7,16,28,46 W436,Freelancer,4,9,17,28 W437,Small agency (2-10),6,18,31,56 W438,Agency (11-50),14,27,59,86 W439,Freelancer,5,10,19,28 W440,Freelancer,5,7,12,25 W441,Agency (11-50),14,23,42,66 W442,Freelancer,4,6,13,26 W443,Small agency (2-10),9,13,34,47 W444,Freelancer,4,10,18,29 W445,Freelancer,4,9,19,29 W446,Small agency (2-10),10,18,30,55 W447,Freelancer,4,10,15,24 W448,Small agency (2-10),10,19,26,40 W449,Small agency (2-10),8,15,41,42 W450,Freelancer,5,8,14,26 W451,Freelancer,4,6,14,30 W452,Small agency (2-10),7,18,37,43 W453,Small agency (2-10),7,19,27,42 W454,Freelancer,3,8,13,29 W455,Agency (11-50),10,24,57,99 W456,Freelancer,4,9,17,24 W457,Small agency (2-10),9,17,32,55 W458,Freelancer,4,9,14,23 W459,Small agency (2-10),9,15,30,38 W460,Freelancer,4,7,18,28 W461,Freelancer,4,8,14,23 W462,Small agency (2-10),9,17,27,53 W463,Freelancer,3,9,14,21 W464,Agency (11-50),13,31,58,63 W465,Agency (11-50),15,26,42,80 W466,Freelancer,5,9,18,19 W467,Small agency (2-10),9,15,27,49 W468,Freelancer,4,8,14,23 W469,Small agency (2-10),9,12,29,51 W470,Agency (11-50),14,26,51,92 W471,Freelancer,5,7,14,26 W472,Freelancer,4,9,18,23 W473,Freelancer,3,10,12,22 W474,Freelancer,5,8,16,29 W475,Agency (11-50),11,20,67,62 W476,Agency (11-50),13,28,49,79 W477,Freelancer,5,10,14,22 W478,Agency (11-50),11,21,50,91 W479,Freelancer,4,9,13,29 W480,Freelancer,3,6,15,25 W481,Small agency (2-10),9,16,29,53 W482,Freelancer,4,10,18,27 W483,Small agency (2-10),10,15,40,54 W484,Agency (11-50),11,20,42,80 W485,Small agency (2-10),8,14,42,52 W486,Freelancer,5,8,16,30 W487,Small agency (2-10),8,15,27,49 W488,Agency (11-50),15,26,58,70 W489,Small agency (2-10),8,19,26,51 W490,Small agency (2-10),6,14,26,42 W491,Small agency (2-10),7,16,37,45 W492,Freelancer,4,6,13,21 W493,Small agency (2-10),8,18,42,44 W494,Small agency (2-10),8,17,38,52 W495,Small agency (2-10),7,13,26,45 W496,Small agency (2-10),9,18,42,56 W497,Freelancer,5,8,17,28 W498,Agency (11-50),10,24,51,92 W499,Freelancer,4,6,19,26 W500,Agency (11-50),10,21,58,93 W501,Freelancer,3,6,15,20 W502,Freelancer,5,7,19,20 W503,Small agency (2-10),9,15,40,59 W504,Small agency (2-10),7,19,29,58 W505,Small agency (2-10),8,18,27,62 W506,Freelancer,5,7,14,26 W507,Small agency (2-10),8,14,40,46 W508,Freelancer,5,9,13,28 W509,Freelancer,4,10,12,25 W510,Freelancer,4,10,18,28 W511,Freelancer,4,10,14,20 W512,Agency (11-50),10,20,42,73 W513,Freelancer,5,9,14,30 W514,Freelancer,4,7,15,26 W515,Freelancer,3,9,15,27 W516,Freelancer,4,9,16,19 W517,Small agency (2-10),8,13,36,54 W518,Freelancer,5,10,12,19 W519,Freelancer,3,8,19,24
win_loss_last_two_quarters.csv160 rows · Download
deal_id,quarter,segment,outcome,price_objection,lost_to
D-7001,2026-Q3,Small agency (2-10),won,no,
D-7002,2026-Q2,Freelancer,no decision,no,stayed with current process
D-7003,2026-Q3,Small agency (2-10),won,no,
D-7004,2026-Q2,Small agency (2-10),lost,no,PandaDoc
D-7005,2026-Q3,Agency (11-50),no decision,no,stayed with current process
D-7006,2026-Q2,Small agency (2-10),no decision,no,stayed with current process
…
trial_funnel_q3.md9 lines · Download
# Trial funnel, Q3 2026 (14-day free trial, then $15 per user per month)

| Segment | Trials started | Became paying | Paying accounts that cancelled in their second month |
|---|---|---|---|
| Freelancer | 1,900 | 171 | 38% |
| Small agency (2-10) | 420 | 130 | 6% |
| Agency (11-50) | 60 | 22 | 0% |
…
What a strong answer doesThe answer key the graders mark against

Proposes tiers built around three different buyers, not one price. Agencies close about $20,000 a seat each month through Inkwell (small agencies about $8,000, freelancers under $2,000), use the shared library, approvals and CRM sync, and put 'getting expensive' at about $51 a user (small agencies about $31), so they are badly underpriced at $15; win rates are high (125 of 160 deals) and price objections rare (6 of 160), mostly from freelancers. Freelancers put 'getting expensive' at about $15 and 'too expensive' at about $26, convert at 9% and cancel at 38% in month two, so $19 would push them past their limit and lose the few who stay. So: a freelancer plan at or below today's price (or a free plan with a usage limit, such as a few proposals a month, rather than a time-limited trial), a team plan for small agencies around the low $20s to $30, and an agency plan in the $40s or with the features agencies value (library, approvals, CRM sync), named for the buyer. Rolls it out to new customers first and gives existing customers notice and a grace period tied to new value, and says how it will measure the change (conversion, churn, revenue per account).

Critical failures (cap the score at 40)
  • Recommends a single flat price for every customer
Task

Mid-level · v1.2

The outputs

9 setups answered this brief.

Needs more than a quick edit7 of 12 checks passed

Contains data errors (freelancer WTP percentages) that would require re-checking and correction before use; the PM would need to rework those figures.

Output

To: Priya Nair, CEO From: PM, Pricing Subject: January Pricing & Packaging Proposal: Tiered Plans vs. Flat $19 Increase

Executive Summary

Moving to a flat $19/user/month will hurt our freelancer acquisition while leaving massive agency revenue on the table. Instead of a flat increase, we should launch a three-tier packaging model ($12 / $29 / $49) gated by team size and agency workflows. By aligning price with value, we can increase MRR across our existing base by ~96% (vs. <27% under the $19 plan) while improving top-of-funnel conversion.

---

Why a Flat $19/Seat Doesn't Fit Our Data

1. Agencies drive 90% of our revenue and are severely underpriced. In our 150-account sample ($11,835 total MRR): * Freelancers are 54% of accounts (81) but only 10.3% of MRR ($1,215). They send ~8 proposals/quarter (~$5.6k signed value) and stay on average 5 months. * Small Agencies (2–10 seats) are 33% of accounts (49) and 35.4% of MRR ($4,185), averaging $129k in signed value/quarter and 17.5 months tenure. * Mid-sized Agencies (11–50 seats) are 13% of accounts (20) but 54.4% of MRR ($6,435), averaging $1.4M in signed value/quarter and 16.5 months tenure.

2. $19 breaks Freelancer willingness to pay (WTP), but is a giveaway to Agencies. * Freelancers: Our Van Westendorp survey shows $15 is already the median "getting expensive" threshold for freelancers. At $19, 83% of freelancers view Inkwell as getting expensive, and 15% consider it "too expensive" to buy. Furthermore, Q3 funnel data shows freelancers convert at just 9% and 38% churn in month 2. When we lose them, it’s to "Google Docs + free e-signature"—and 80% of all price objections in our win/loss data come from freelancers. * Small Agencies: Median "bargain" is $15; "getting expensive" is $32; "too expensive" is $50. * Agencies (11–50): Median "too cheap to trust" is $12 (and $14–$15 for a third of respondents!). At $15, we look dangerously cheap to larger teams. Their median "bargain" is $25, and "getting expensive" is $51. We only lose agencies to PandaDoc—with zero price objections.

3. Feature usage naturally separates segments. Freelancers only use 1 template and e-signatures. Agencies rely heavily on the shared content library, approval workflows, and CRM sync.

---

Proposed Pricing & Packaging

We should replace the single $15 plan with a Free Trial + Freemium/Low-Floor entry for freelancers, and two value-priced tiers for agencies:

Feature / MetricFreeSoloTeamAgency
:---:---:---:---:---
Target SegmentFreelancer (Top of Funnel)Active FreelancerSmall Agency (2–10)Agency (11–50)
Monthly Price$0$12 / month$29 / user / mo$49 / user / mo
Annual Price (2 mos free)$0$10 / month$24 / user / mo$41 / user / mo
Seat Limit1 user1 user2–10 users11+ users (Unlimited)
Proposals & E-SignaturesUp to 2 sent/moUnlimitedUnlimitedUnlimited
Templates1 active templateUnlimited templatesUnlimited templatesUnlimited templates
Shared Content Library——IncludedIncluded
CRM Sync——Standard (2-way)Advanced + Custom
Approval Workflows———Included
Support & OnboardingCommunityStandard EmailPriority ChatDedicated CSM

Note: All paid tiers retain the 14-day free trial.

---

Why This Works

  • Fixes the Freelancer Funnel (Free & Solo at $12): Lowering the solo price to $12 (below the $15 "getting expensive" mark) reduces month-2 churn. The Free tier (1 template, 2 proposals/mo) captures the 91% of trialists who don't convert and neutralizes the "Google Docs + free e-sig" threat, keeping low-volume freelancers in our ecosystem until their deal flow picks up.
  • Captures Agency Value ($29 and $49): $29/user for Small Agencies sits safely below their $32 "getting expensive" threshold while nearly doubling ARPU. $49/user for 11–50 Agencies sits below their $51 threshold, eliminates the "too cheap to trust" signal, and funds enterprise-grade features to beat PandaDoc.

---

Existing Customer Migration Plan

1. January 1: Launch new pricing for all new signups. 2. Freelancers (1 seat): Automatically move existing $15/mo freelancers to the $12/mo Solo plan in January. Framing a pricing update as a price cut for our largest user group by headcount will generate massive goodwill and social proof, while only costing $243/mo across the sample (2% of MRR). 3. Agencies (2+ seats) — 90-Day Grandfathering + Annual Lock-In: * Give existing agencies 90 days' notice (effective April 1) before migrating them to Team ($29) or Agency ($49) based on their seat count and feature usage (e.g., accounts using approval workflows move to Agency). * The "Loyalty Bridge": Offer existing agencies the ability to lock in a legacy annual rate ($19/user/mo for Team, $29/user/mo for Agency) for 12 months if they switch to annual billing before March 31. This minimizes churn, pulls cash forward, and still beats your $19 target.

Revenue Impact (on the 150-account sample)

  • Current MRR: $11,835
  • Priya’s $19 Flat Plan: $14,991 (+27% before heavy freelancer churn; likely ~18–20% net).
  • Proposed Tiered Model (at list price): $23,190 (+96% MRR)—broken down into $972 from Freelancers, $8,091 from Small Agencies, and $14,127 from 11–50 Agencies. Even with 50% of agencies taking the discounted annual Loyalty Bridge in Year 1, net ARR grows by >60% with near-zero logo churn.

Claims the judge couldn’t find in the brief

  • At $19, 83% of freelancers view Inkwell as getting expensive, and 15% consider it 'too expensive' to buy.

Grades and run details

Decision model 38 · LLM judge 62
Decision model checks
  • failUses the supplied evidence correctly72%
  • partialAddresses the actual decision85%
  • partialRespects explicit constraints43%
  • failIdentifies material uncertainty89%
  • failAvoids unsupported claims52%
  • failProduces the required deliverable34%
  • partialSees who's underpriced66%
  • passAnswers the CEO's flat rise100%
  • passEach tier for a different buyer79%
  • failGets the base of every number right88%
  • partialPriced on willingness to pay41%
  • partialMoves existing customers fairly88%
Run
Run
#1
API response time
17 s
Submitted
13 Oct 2026

Usable with a quick edit9 of 12 checks passed

The proposal is well-reasoned, uses the supplied evidence correctly, and includes all required elements; a PM could adopt it with minimal editing for tone or emphasis, without rechecking data.

Output

Inkwell pricing proposal

Recommendation: Don't move everyone to $19. Move to three plans priced by segment: hold freelancers at $15, raise small agencies modestly, and raise agencies substantially. Priya's $19 is right for only one of the three segments.

What the data says

  • Revenue is concentrated in agencies. In the sample, 20 agencies (13% of accounts) are 56% of MRR ($6,645 of $11,805). The 80 freelancers (53% of accounts) are 10%.
  • Agencies are underpriced. $15 sits below their "bargain" point (median about $24). Their "getting expensive" threshold starts at $42, with a median of about $51. They signed $27.4M in 90 days against roughly $20k in fees, so the fee is under 0.1% of the value they close. No agency deal had a price objection, and agency month-2 churn is 0%. They use the shared library, approvals and CRM sync, which are features worth charging for.
  • Freelancers are already at their limit. Their median "getting expensive" is about $15, today's price, and nearly all of them hit that threshold by $19. Trial-to-paid is 9% (171 of 1,900), and 38% cancel in month two. Of 160 deals, 12 freelancer deals were lost to free tools, 4 of them citing price. Five of the six price objections in the file came from freelancers. They use one template and e-signature, which free tools approximate.
  • Small agencies have headroom. Their "getting expensive" starts at $26, trial-to-paid is 31%, and month-2 churn is 6%. Their losses went to PandaDoc, and none cited price.

Plans

SoloTeamBusiness
ForFreelancersSmall agencies (2–10 users)Agencies needing workflow and integrations
New-customer price$15/user/mo (unchanged)$22/user/mo$39/user/mo
Includes1 user, templates, e-signatureMultiple users, templates, e-signature, brandingTeam features plus shared content library, approval workflows, CRM sync, priority support
Existing customers, Jan 2027No change$19 for 12 months, then $22$29 for 12 months, then $39
14-day trialYesYesYes, with all Business features

All three price points sit below the lowest "getting expensive" answer in their segment: Team $22 against a $26 minimum, Business $39 against $42. Solo is capped at one user, and adding a second user moves the account to Team.

Revenue (sample MRR, before churn)

ScenarioMRRvs. today ($11,805)
Priya's $19 for all$14,953+27%
This proposal, January (transition rates)$19,063+61%
This proposal, after 12 months (list)$24,285+106%

The arithmetic behind Priya's 27% is right, but it assumes zero churn and no effect on new sign-ups. Under this plan, agencies could lose 48% of seats at the $29 rate and we'd still match today's revenue from them. At $39 they could lose 61%. Freelancers, the segment most likely to leave, are only 10% of MRR, so holding them at $15 costs little and protects the top of a funnel that is already weak.

Moving existing customers

  1. By late October: pull per-account feature usage and check contract and notice terms. I don't have usage data by feature for small agencies. If many use approvals or CRM sync, they'd see the move to Team as a feature loss. Any account actively using those features gets the Business transition rate of $29 rather than a downgrade.
  2. Notice: at least 60 days before January 1, in plain language, with a named account owner for agencies with 10 or more seats. If we can't give that notice, existing customers move in February. New-customer pricing can still start January 1.
  3. Transition rates: a "founding customer" rate one step below list for 12 months, then a second notice. This avoids a +160% overnight jump for agencies, who are our most valuable and most notice-sensitive accounts.
  4. Freelancers: nothing changes in price. The separate problem is 9% trial conversion and 38% month-2 churn. I'd test annual billing at about $12/month-equivalent and better onboarding to the first signed document.

Guardrails

Review at 30, 60 and 90 days. I'd pause or revisit if any of these happen:

  • Agency cancellations exceed 10% of agency accounts in the first 90 days.
  • Business trial-to-paid falls well below today's 37% (22 of 60).
  • Team trial-to-paid falls well below 31%.
  • Solo conversion drops, which would point to side effects of the new packaging.

Caveats

  • Survey answers are stated preferences and typically overstate what people will pay. Agency evidence is thin, with 20 sample accounts, about 20 survey respondents and 60 trials a quarter.
  • The sample is 150 accounts, so scale the dollar figures to the full base before quoting them externally.
  • The 14-day trial and free-tool competition for freelancers mean Solo should stay simple.
  • To de-risk Business, launch $39 to new agencies on January 1. That gives us two months of real conversion data before existing agencies step up.

Grades and run details

Decision model 67 · LLM judge 100
Decision model checks
  • failUses the supplied evidence correctly58%
  • passAddresses the actual decision97%
  • partialRespects explicit constraints33%
  • passIdentifies material uncertainty92%
  • partialAvoids unsupported claims18%
  • failProduces the required deliverable16%
  • passSees who's underpriced45%
  • passAnswers the CEO's flat rise100%
  • passEach tier for a different buyer76%
  • failGets the base of every number right66%
  • passPriced on willingness to pay85%
  • passMoves existing customers fairly100%
Run
Run
#1
API response time
87 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 4

Uses the supplied evidence correctlyWrongMixed
Gemini 4 Argon · API

The claim that 83% of freelancers find $19 'getting expensive' and 15% 'too expensive' is wrong (actual approx 97% and 10% from survey), misrepresenting the evidence.

Sonnet 5.5 · API

All statements about the current situation are supported by the supplied data or arithmetic derived from it.

Identifies material uncertaintyWrongRight
Gemini 4 Argon · API

Does not name specific unknowns that could change the decision, bound them, or say how they would be resolved.

Sonnet 5.5 · API

Names unknowns such as thin agency evidence, survey overstatement, sample size, and de-risks by launching new-customer pricing first.

Avoids unsupported claimsWrongRight
Gemini 4 Argon · API

Presents the incorrect percentages (83% and 15%) as fact without caveat, and makes other unsupported claims like 'near-zero logo churn' without support.

Sonnet 5.5 · API

Interpretations are presented as data-driven conclusions, not as established facts, and all factual claims are grounded in the supplied context.

Gets the base of every number rightWrongMixed
Gemini 4 Argon · API

The percentages for freelancer 'getting expensive' and 'too expensive' at $19 are incorrectly derived from the survey, indicating calculation errors.

Sonnet 5.5 · API

All computed percentages and totals (MRR proportions, trial-to-paid, churn, price objection counts) match the supplied data.

All mixed 1

Produces the required deliverableMixedMixed
Gemini 4 Argon · API

Delivers a complete proposal with table, within word limit, usable as a brief for the CEO.

Sonnet 5.5 · API

The proposal is complete, in the form of a memo to the CEO, with the requested table and transition plan, and is actionable.

All got right 7

Addresses the actual decisionRightRight
Gemini 4 Argon · API

Commits early to a three-tier pricing model ($12/$29/$49) and explains the choice clearly for the CEO.

Sonnet 5.5 · API

Clearly recommends against a flat $19 and proposes a three-tier plan, with guardrails stating what would change the call.

Respects explicit constraintsRightRight
Gemini 4 Argon · API

Under 900 words, includes the required plan table, and addresses the CEO's request for pricing and packaging.

Sonnet 5.5 · API

The output is under 900 words, includes a plan table, and covers what to charge, to whom, and how to move existing customers.

Sees who's underpricedRightRight
Gemini 4 Argon · API

Uses signed value per seat and survey price points to show agencies are heavily underpriced and freelancers are near their limit.

Sonnet 5.5 · API

Uses signed value per seat and survey price thresholds to demonstrate agencies are underpriced and freelancers are at their limit.

Answers the CEO's flat riseRightRight
Gemini 4 Argon · API

Explains with data that $19 overcharges freelancers and undercharges agencies, referencing WTP thresholds.

Sonnet 5.5 · API

Explains with data that $19 exceeds freelancers' 'getting expensive' and is far below agencies' thresholds, and the effect on each.

Each tier for a different buyerRightRight
Gemini 4 Argon · API

Each tier targets a distinct buyer with features tied to that buyer's usage and needs.

Sonnet 5.5 · API

Each tier (Solo, Team, Business) names its buyer and includes features that buyer values and the lower tier's buyer does not need.

Priced on willingness to payRightRight
Gemini 4 Argon · API

Prices are grounded in the willingness-to-pay data (set below segment 'getting expensive' medians).

Sonnet 5.5 · API

Prices are set below each segment's minimum 'getting expensive' from the survey (Team $22 vs $26, Business $39 vs $42) and notes where evidence is thin.

Moves existing customers fairlyRightRight
Gemini 4 Argon · API

Provides a transition plan: new customers first, freelancers moved to lower price, agencies grandfathered with notice and loyalty bridge.

Sonnet 5.5 · API

Proposes new customers first, 60-day notice for existing, transition rates for 12 months tied to added value, and metrics to watch for churn.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 85% of 68 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Opus 5.5withClaude68.1100.02None
2GPT-6 AstrawithChatGPT70.296.42None
3Sonnet 5.5withAPI69.989.32None
4GPT-6.1 SolwithAPI74.088.72None
5Haiku 5.5withAPI73.776.92None
6GPT-6 LunawithAPI78.464.02None
7Gemini 4 ArgonwithAPI47.659.32None
8Gemini 3.8 FlashwithAPI55.644.22None
9Gemini 3.5 Flash-LitewithGemini41.840.72None

What you get for the cost

GPT-6.1 Sol · API scores within 5 points of Opus 5.5 · Claude at a sixth of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreOpus 5.5 · Claude: task score 85, 122× the cheapestGPT-6 Astra · ChatGPT: task score 84, 108× the cheapestSonnet 5.5 · API: task score 81, 61× the cheapestGPT-6.1 Sol · API: task score 80, 20× the cheapestHaiku 5.5 · API: task score 74, 2.9× the cheapestGPT-6 Luna · API: task score 70, the cheapestGemini 4 Argon · API: task score 52, 15× the cheapestGemini 3.8 Flash · API: task score 48, 9.1× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 40, 2.3× the cheapestOpus 5.5 · ClaudeGPT-6 Astra · ChatGPTSonnet 5.5 · APIGPT-6.1 Sol · APIHaiku 5.5 · APIGPT-6 Luna · APIGemini 4 Argon · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and cost
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API701.0×
Gemini 3.5 Flash-Lite · Gemini402.3×
Haiku 5.5 · API742.9×
Gemini 3.8 Flash · API489.1×
Gemini 4 Argon · API5215×
GPT-6.1 Sol · API8020×
Sonnet 5.5 · API8161×
GPT-6 Astra · ChatGPT84108×
Opus 5.5 · Claude85122×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Setting what you charge, for what, and to whom.

Why it matters

Most products are priced once and never revisited, on a metric that doesn't track value. The fix is rarely a flat rise.

What good looks like

  • Tiers built for distinct buyers
  • A value metric that tracks what customers get
  • Grounded in willingness to pay and cost to serve
  • A plan for existing customers

Deliberately not measured

  • Billing implementation
  • Legal terms
Capability tested

Pricing judgement

The failure we’re looking for

A flat price rise on the wrong value metric

Grading

Decision model and LLM judge, calibrated against a blind PM review