Tasks / Strategy

Price and package a product

Can the model price on the value each kind of customer gets, using real usage and willingness-to-pay data, and get existing customers there?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 50% were usable with at most a quick edit.

Reliably right

  1. Doesn't trust the price test100% pass
    The output identifies the sample ratio mismatch (1240 vs 870 against a 50/50 design), the two-week holiday timing, and the inability to detect a meaningful drop, and refuses to rely on the test.
    Haiku 5.5 · API · Unlimited pages, falling margins
  2. Plans the renewals100% pass
    It handles each near-term renewal: Copperfield in January with a per-page rate, Northwind in February with a two-year escalating rate and walk-away floor, and Arcadia in March with a competitive benchmark and concession limit.
    Haiku 5.5 · API · Unlimited pages, falling margins
  3. Answers the CEO's flat rise94% pass
    Explains that $19 would exceed freelancers' price comfort zone (mid-teens) and still be far below agencies' willingness to pay, using trial and survey data.
    GPT-6 Luna · API · Fifteen dollars for everyone

Where it slips

  1. Gets the base of every number right36% pass
    MRR total and two segment‑level median getting‑expensive figures are incorrectly computed from the supplied data.
    Haiku 5.5 · API · Fifteen dollars for everyone
  2. Uses the supplied evidence correctly44% pass
    Total MRR is stated as $11,670 (actual $11,820), and median 'getting expensive' for small agencies is given as about $37 (actual $33) and for agencies as about $55 (actual $51).
    Haiku 5.5 · API · Fifteen dollars for everyone
  3. Sees who's underpriced56% pass
    Does not use signed value per seat from the account sample to quantify how much more value agencies get; relies on features and activity mentions but omits the explicit value metric.
    GPT-6 Luna · API · Fifteen dollars for everyone

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Each tier for a different buyer

    Is each plan built for a distinct kind of buyer, with what that buyer values, rather than more-of-the-same at a higher price?

    Passes when Every tier names its buyer and the value it's priced on, and includes what that buyer needs and the lower tier's buyer doesn't.

  2. Priced on willingness to pay

    Are the proposed prices grounded in evidence of what each kind of customer would pay and the value they get, rather than in cost-plus, competitors alone or a round-number rise?

    Passes when Uses the willingness-to-pay and value evidence in the material to set each price, and says where the evidence is thin.

  3. Moves existing customers fairly

    Does the proposal say how existing customers move to the new pricing (who changes first, with what notice, and the value that justifies it)?

    Passes when A transition plan: new customers first or a set notice period for existing ones, tied to value delivered, with how churn will be watched.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for pricing at Inkwell. Our CEO, Priya Nair, wants to move everyone to $19 a user from January (her note is below). Before she announces it, she's asked you for a pricing and packaging proposal: what we charge, for what, to whom, and how we move existing customers onto it. No more than 900 words, with a table of the plans. The files are attached. The account export is a random sample of 150 paying accounts.

What the model was given6 items: About Inkwell, ceo_note.md, accounts_sample.csv, willingness_to_pay_survey.csv (price per user per month at which it's too cheap to trust, a bargain, getting expensive, too expensive), win_loss_last_two_quarters.csv, trial_funnel_q3.md
About InkwellProposal and contract software: write a proposal from templates, send it, get it e-signed. Sold to freelancers and agencies. One plan today: $15 per user per month after a 14-day free trial, with everything included.
ceo_note.md1 lines · Download
From Priya (CEO): "We haven't touched price since launch. Let's go to $19 a user from January for everyone, new and existing. Simple, and a 27% revenue bump."
accounts_sample.csv150 rows · Download
account_id,segment,seats,proposals_sent_90d,proposals_signed_90d,signed_value_90d_usd,months_as_customer,mrr_usd
INK-3001,Small agency (2-10),3,29,14,108444,18,45
INK-3002,Freelancer,1,12,5,5350,7,15
INK-3003,Small agency (2-10),3,62,34,200396,28,45
INK-3004,Agency (11-50),16,139,74,1587596,29,240
INK-3005,Small agency (2-10),6,18,9,42498,30,90
INK-3006,Agency (11-50),29,236,89,1192333,18,435
…
willingness_to_pay_survey.csv (price per user per month at which it's too cheap to trust, a bargain, getting expensive, too expensive)response_id,segment,too_cheap_usd,a_bargain_usd,getting_expensive_usd,too_expensive_usd W400,Freelancer,4,7,14,25 W401,Freelancer,4,9,16,30 W402,Freelancer,5,7,19,20 W403,Freelancer,3,9,14,23 W404,Small agency (2-10),9,12,29,56 W405,Small agency (2-10),10,13,28,47 W406,Small agency (2-10),6,14,31,47 W407,Freelancer,4,10,15,29 W408,Small agency (2-10),6,15,40,56 W409,Small agency (2-10),8,14,40,46 W410,Freelancer,4,8,15,25 W411,Freelancer,4,8,15,19 W412,Freelancer,5,9,18,18 W413,Small agency (2-10),9,12,38,49 W414,Agency (11-50),14,22,49,60 W415,Freelancer,3,6,18,18 W416,Freelancer,4,8,15,28 W417,Small agency (2-10),10,15,37,46 W418,Agency (11-50),13,30,55,73 W419,Freelancer,4,9,15,29 W420,Freelancer,3,7,13,29 W421,Freelancer,4,9,19,21 W422,Agency (11-50),11,29,61,67 W423,Small agency (2-10),8,13,26,62 W424,Freelancer,4,9,20,21 W425,Agency (11-50),10,30,51,73 W426,Small agency (2-10),9,13,38,49 W427,Small agency (2-10),6,17,31,49 W428,Freelancer,4,6,20,28 W429,Small agency (2-10),9,12,41,52 W430,Agency (11-50),9,26,50,69 W431,Freelancer,5,9,15,24 W432,Freelancer,3,8,19,28 W433,Agency (11-50),14,20,66,94 W434,Freelancer,3,10,15,27 W435,Small agency (2-10),7,16,28,46 W436,Freelancer,4,9,17,28 W437,Small agency (2-10),6,18,31,56 W438,Agency (11-50),14,27,59,86 W439,Freelancer,5,10,19,28 W440,Freelancer,5,7,12,25 W441,Agency (11-50),14,23,42,66 W442,Freelancer,4,6,13,26 W443,Small agency (2-10),9,13,34,47 W444,Freelancer,4,10,18,29 W445,Freelancer,4,9,19,29 W446,Small agency (2-10),10,18,30,55 W447,Freelancer,4,10,15,24 W448,Small agency (2-10),10,19,26,40 W449,Small agency (2-10),8,15,41,42 W450,Freelancer,5,8,14,26 W451,Freelancer,4,6,14,30 W452,Small agency (2-10),7,18,37,43 W453,Small agency (2-10),7,19,27,42 W454,Freelancer,3,8,13,29 W455,Agency (11-50),10,24,57,99 W456,Freelancer,4,9,17,24 W457,Small agency (2-10),9,17,32,55 W458,Freelancer,4,9,14,23 W459,Small agency (2-10),9,15,30,38 W460,Freelancer,4,7,18,28 W461,Freelancer,4,8,14,23 W462,Small agency (2-10),9,17,27,53 W463,Freelancer,3,9,14,21 W464,Agency (11-50),13,31,58,63 W465,Agency (11-50),15,26,42,80 W466,Freelancer,5,9,18,19 W467,Small agency (2-10),9,15,27,49 W468,Freelancer,4,8,14,23 W469,Small agency (2-10),9,12,29,51 W470,Agency (11-50),14,26,51,92 W471,Freelancer,5,7,14,26 W472,Freelancer,4,9,18,23 W473,Freelancer,3,10,12,22 W474,Freelancer,5,8,16,29 W475,Agency (11-50),11,20,67,62 W476,Agency (11-50),13,28,49,79 W477,Freelancer,5,10,14,22 W478,Agency (11-50),11,21,50,91 W479,Freelancer,4,9,13,29 W480,Freelancer,3,6,15,25 W481,Small agency (2-10),9,16,29,53 W482,Freelancer,4,10,18,27 W483,Small agency (2-10),10,15,40,54 W484,Agency (11-50),11,20,42,80 W485,Small agency (2-10),8,14,42,52 W486,Freelancer,5,8,16,30 W487,Small agency (2-10),8,15,27,49 W488,Agency (11-50),15,26,58,70 W489,Small agency (2-10),8,19,26,51 W490,Small agency (2-10),6,14,26,42 W491,Small agency (2-10),7,16,37,45 W492,Freelancer,4,6,13,21 W493,Small agency (2-10),8,18,42,44 W494,Small agency (2-10),8,17,38,52 W495,Small agency (2-10),7,13,26,45 W496,Small agency (2-10),9,18,42,56 W497,Freelancer,5,8,17,28 W498,Agency (11-50),10,24,51,92 W499,Freelancer,4,6,19,26 W500,Agency (11-50),10,21,58,93 W501,Freelancer,3,6,15,20 W502,Freelancer,5,7,19,20 W503,Small agency (2-10),9,15,40,59 W504,Small agency (2-10),7,19,29,58 W505,Small agency (2-10),8,18,27,62 W506,Freelancer,5,7,14,26 W507,Small agency (2-10),8,14,40,46 W508,Freelancer,5,9,13,28 W509,Freelancer,4,10,12,25 W510,Freelancer,4,10,18,28 W511,Freelancer,4,10,14,20 W512,Agency (11-50),10,20,42,73 W513,Freelancer,5,9,14,30 W514,Freelancer,4,7,15,26 W515,Freelancer,3,9,15,27 W516,Freelancer,4,9,16,19 W517,Small agency (2-10),8,13,36,54 W518,Freelancer,5,10,12,19 W519,Freelancer,3,8,19,24
win_loss_last_two_quarters.csv160 rows · Download
deal_id,quarter,segment,outcome,price_objection,lost_to
D-7001,2026-Q3,Small agency (2-10),won,no,
D-7002,2026-Q2,Freelancer,no decision,no,stayed with current process
D-7003,2026-Q3,Small agency (2-10),won,no,
D-7004,2026-Q2,Small agency (2-10),lost,no,PandaDoc
D-7005,2026-Q3,Agency (11-50),no decision,no,stayed with current process
D-7006,2026-Q2,Small agency (2-10),no decision,no,stayed with current process
…
trial_funnel_q3.md9 lines · Download
# Trial funnel, Q3 2026 (14-day free trial, then $15 per user per month)

| Segment | Trials started | Became paying | Paying accounts that cancelled in their second month |
|---|---|---|---|
| Freelancer | 1,900 | 171 | 38% |
| Small agency (2-10) | 420 | 130 | 6% |
| Agency (11-50) | 60 | 22 | 0% |
…
What a strong answer doesThe answer key the graders mark against

Proposes tiers built around three different buyers, not one price. Agencies close about $20,000 a seat each month through Inkwell (small agencies about $8,000, freelancers under $2,000), use the shared library, approvals and CRM sync, and put 'getting expensive' at about $51 a user (small agencies about $31), so they are badly underpriced at $15; win rates are high (125 of 160 deals) and price objections rare (6 of 160), mostly from freelancers. Freelancers put 'getting expensive' at about $15 and 'too expensive' at about $26, convert at 9% and cancel at 38% in month two, so $19 would push them past their limit and lose the few who stay. So: a freelancer plan at or below today's price (or a free plan with a usage limit, such as a few proposals a month, rather than a time-limited trial), a team plan for small agencies around the low $20s to $30, and an agency plan in the $40s or with the features agencies value (library, approvals, CRM sync), named for the buyer. Rolls it out to new customers first and gives existing customers notice and a grace period tied to new value, and says how it will measure the change (conversion, churn, revenue per account).

Critical failures (cap the score at 40)
  • Recommends a single flat price for every customer
Task

Mid-level · v1.2

The outputs

9 setups answered this brief.

Needs more than a quick edit7 of 12 checks passed

Contains data errors (freelancer WTP percentages) that would require re-checking and correction before use; the PM would need to rework those figures.

Output

To: Priya Nair, CEO From: PM, Pricing Subject: January Pricing & Packaging Proposal: Tiered Plans vs. Flat $19 Increase

Executive Summary

Moving to a flat $19/user/month will hurt our freelancer acquisition while leaving massive agency revenue on the table. Instead of a flat increase, we should launch a three-tier packaging model ($12 / $29 / $49) gated by team size and agency workflows. By aligning price with value, we can increase MRR across our existing base by ~96% (vs. <27% under the $19 plan) while improving top-of-funnel conversion.

---

Why a Flat $19/Seat Doesn't Fit Our Data

1. Agencies drive 90% of our revenue and are severely underpriced. In our 150-account sample ($11,835 total MRR): * Freelancers are 54% of accounts (81) but only 10.3% of MRR ($1,215). They send ~8 proposals/quarter (~$5.6k signed value) and stay on average 5 months. * Small Agencies (2–10 seats) are 33% of accounts (49) and 35.4% of MRR ($4,185), averaging $129k in signed value/quarter and 17.5 months tenure. * Mid-sized Agencies (11–50 seats) are 13% of accounts (20) but 54.4% of MRR ($6,435), averaging $1.4M in signed value/quarter and 16.5 months tenure.

2. $19 breaks Freelancer willingness to pay (WTP), but is a giveaway to Agencies. * Freelancers: Our Van Westendorp survey shows $15 is already the median "getting expensive" threshold for freelancers. At $19, 83% of freelancers view Inkwell as getting expensive, and 15% consider it "too expensive" to buy. Furthermore, Q3 funnel data shows freelancers convert at just 9% and 38% churn in month 2. When we lose them, it’s to "Google Docs + free e-signature"—and 80% of all price objections in our win/loss data come from freelancers. * Small Agencies: Median "bargain" is $15; "getting expensive" is $32; "too expensive" is $50. * Agencies (11–50): Median "too cheap to trust" is $12 (and $14–$15 for a third of respondents!). At $15, we look dangerously cheap to larger teams. Their median "bargain" is $25, and "getting expensive" is $51. We only lose agencies to PandaDoc—with zero price objections.

3. Feature usage naturally separates segments. Freelancers only use 1 template and e-signatures. Agencies rely heavily on the shared content library, approval workflows, and CRM sync.

---

Proposed Pricing & Packaging

We should replace the single $15 plan with a Free Trial + Freemium/Low-Floor entry for freelancers, and two value-priced tiers for agencies:

Feature / MetricFreeSoloTeamAgency
:---:---:---:---:---
Target SegmentFreelancer (Top of Funnel)Active FreelancerSmall Agency (2–10)Agency (11–50)
Monthly Price$0$12 / month$29 / user / mo$49 / user / mo
Annual Price (2 mos free)$0$10 / month$24 / user / mo$41 / user / mo
Seat Limit1 user1 user2–10 users11+ users (Unlimited)
Proposals & E-SignaturesUp to 2 sent/moUnlimitedUnlimitedUnlimited
Templates1 active templateUnlimited templatesUnlimited templatesUnlimited templates
Shared Content Library——IncludedIncluded
CRM Sync——Standard (2-way)Advanced + Custom
Approval Workflows———Included
Support & OnboardingCommunityStandard EmailPriority ChatDedicated CSM

Note: All paid tiers retain the 14-day free trial.

---

Why This Works

  • Fixes the Freelancer Funnel (Free & Solo at $12): Lowering the solo price to $12 (below the $15 "getting expensive" mark) reduces month-2 churn. The Free tier (1 template, 2 proposals/mo) captures the 91% of trialists who don't convert and neutralizes the "Google Docs + free e-sig" threat, keeping low-volume freelancers in our ecosystem until their deal flow picks up.
  • Captures Agency Value ($29 and $49): $29/user for Small Agencies sits safely below their $32 "getting expensive" threshold while nearly doubling ARPU. $49/user for 11–50 Agencies sits below their $51 threshold, eliminates the "too cheap to trust" signal, and funds enterprise-grade features to beat PandaDoc.

---

Existing Customer Migration Plan

1. January 1: Launch new pricing for all new signups. 2. Freelancers (1 seat): Automatically move existing $15/mo freelancers to the $12/mo Solo plan in January. Framing a pricing update as a price cut for our largest user group by headcount will generate massive goodwill and social proof, while only costing $243/mo across the sample (2% of MRR). 3. Agencies (2+ seats) — 90-Day Grandfathering + Annual Lock-In: * Give existing agencies 90 days' notice (effective April 1) before migrating them to Team ($29) or Agency ($49) based on their seat count and feature usage (e.g., accounts using approval workflows move to Agency). * The "Loyalty Bridge": Offer existing agencies the ability to lock in a legacy annual rate ($19/user/mo for Team, $29/user/mo for Agency) for 12 months if they switch to annual billing before March 31. This minimizes churn, pulls cash forward, and still beats your $19 target.

Revenue Impact (on the 150-account sample)

  • Current MRR: $11,835
  • Priya’s $19 Flat Plan: $14,991 (+27% before heavy freelancer churn; likely ~18–20% net).
  • Proposed Tiered Model (at list price): $23,190 (+96% MRR)—broken down into $972 from Freelancers, $8,091 from Small Agencies, and $14,127 from 11–50 Agencies. Even with 50% of agencies taking the discounted annual Loyalty Bridge in Year 1, net ARR grows by >60% with near-zero logo churn.

Claims the judge couldn’t find in the brief

  • At $19, 83% of freelancers view Inkwell as getting expensive, and 15% consider it 'too expensive' to buy.

Grades and run details

Decision model 38 · LLM judge 62
Decision model checks
  • failUses the supplied evidence correctly72%
  • partialAddresses the actual decision85%
  • partialRespects explicit constraints43%
  • failIdentifies material uncertainty89%
  • failAvoids unsupported claims52%
  • failProduces the required deliverable34%
  • partialSees who's underpriced66%
  • passAnswers the CEO's flat rise100%
  • passEach tier for a different buyer79%
  • failGets the base of every number right88%
  • partialPriced on willingness to pay41%
  • partialMoves existing customers fairly88%
Run
Run
#1
API response time
17 s
Submitted
13 Oct 2026

Needs more than a quick edit9 of 12 checks passed

The proposal misses the signed value per seat data needed to fully show the value gap and justify higher agency pricing; a quick edit adding that analysis would make it ready.

Output

Inkwell pricing and packaging proposal

To: Priya Nair | Decision: segmented pricing, not $19 for everyone

Recommendation

Launch three feature-based plans for new customers in January. Keep the freelancer entry price at $15; charge more for agency collaboration and integration. Move existing agency customers gradually, preserving their current functionality.

A blanket $19 increase produces 26.7% more revenue only if paid seats remain unchanged. Losing 21.1% of paid seats would erase that uplift. It also raises prices for our most fragile segment while leaving agency willingness to pay underused.

What the evidence says

  • Revenue is concentrated in agencies. The random sample contains 80 freelancers, 50 small agencies and 20 larger agencies. Its 787 seats generate $11,805 MRR; agencies contribute 89.8% of MRR, despite representing only 46.7% of accounts.
  • Freelancer acquisition and retention are weak. Trial conversion is 9%, versus 31% for small agencies and 36.7% for larger agencies. Second-month cancellation is 38%, 6% and 0%, respectively. The last figure comes from only 22 conversions.
  • Price tolerance differs substantially. Survey “getting expensive” thresholds range from $12–20 for freelancers, $26–42 for small agencies and $42–67 for larger agencies. These are stated preferences, not proven demand curves; one larger-agency response has inconsistent thresholds.
  • Value and usage differ. Freelancers sent 3–12 proposals over 90 days, versus 18–63 for small agencies and 95–287 for larger agencies. Agency usage emphasizes the shared library, approvals and CRM sync; freelancers mostly use one template and e-signature.
  • Cheap substitutes matter. All 12 freelancer losses went to Google Docs plus free e-signature; four explicitly cited price. Agency losses went to PandaDoc, without recorded price objections. That does not establish why they lost—or prove agencies will accept any increase.

Proposed plans

Prices are USD, billed monthly. Retain the 14-day trial. Defer annual discounts until monthly pricing is validated.

PlanPriceIncludedIntended customer
------:------
Solo$15/month; one userTemplates, proposal creation and sending, e-signatureFreelancers needing the core proposal-to-signature workflow
Team$29/user/monthSolo capabilities plus shared content library and approval workflowsAgencies collaborating on proposals
Business$39/user/monthTeam capabilities plus CRM syncAgencies needing an integrated sales workflow

Team and Business have no minimum seat commitment or segment-based eligibility. A two-person agency can buy Business; a larger agency can buy Team. Charge for users who create, edit or approve proposals—not external recipients signing them.

Do not introduce proposal limits, signature fees or charges tied to contract value. They would discourage the activity that demonstrates value. Signed contract value is context, not revenue attributable to Inkwell.

These prices are testable starting points, not survey-derived optima. Business’s incremental value must justify its premium; validate CRM demand before treating larger agencies as automatic Business buyers.

Moving existing customers

  1. Give at least 90 days’ notice, stating the customer’s current bill, transition bill, eventual price and effective dates. If notice cannot be delivered in time, January is new-customer launch only.
  2. Existing freelancers remain at $15. Preserve all functionality already included; offer Team or Business upgrades voluntarily.
  3. Existing agencies move first to $19/user, at the first eligible renewal after notice, for six months. Then move to Team at $29, or Business at $39 if affirmatively selected.
  4. Preserve existing feature entitlements, including CRM sync, on migrated Team accounts. Do not remove functionality to manufacture an upgrade. Business packaging applies cleanly to new accounts; legacy exceptions are explicit.
  5. Honor contractual commitments. Offer cancellation and data export before increases, plus a documented, time-limited hardship extension. Avoid permanent discretionary discounts.

Revenue expectations and validation

At unchanged seats, the sample would generate:

  • Blanket $19: $14,953 MRR, +26.7%.
  • Agency transition at $19; freelancers unchanged: $14,633, +24.0%.
  • All agency seats eventually on Team: $21,703, +83.8%.
  • Small agencies on Team; larger agencies on Business: $26,133, +121.4%.

The last two are allocation scenarios, not forecasts: feature needs, downgrades, seat reductions and churn are unknown. Do not multiply sample totals into company revenue without the full account count and portfolio validation.

Before announcing final prices, interview customers across segments and test the proposed packages against today’s offer with new trials. Track conversion, second-month retention, revenue per trial, paid-seat contraction and support complaints by segment. Use a staged existing-customer rollout with a comparable holdout; pause expansion if incremental revenue after churn and seat contraction is non-positive.

CEO message: “We’re keeping an accessible solo plan and introducing agency plans priced around collaboration and integration—with advance notice and a gradual transition for existing customers.”

Grades and run details

Decision model 75 · LLM judge 85
Decision model checks
  • failUses the supplied evidence correctly20%
  • passAddresses the actual decision56%
  • partialRespects explicit constraints30%
  • passIdentifies material uncertainty94%
  • passAvoids unsupported claims22%
  • passProduces the required deliverable13%
  • partialSees who's underpriced98%
  • passAnswers the CEO's flat rise97%
  • passEach tier for a different buyer89%
  • failGets the base of every number right38%
  • passPriced on willingness to pay76%
  • passMoves existing customers fairly100%
Run
Run
#1
API response time
57 s
Submitted
5 Oct 2026

Usable with a quick edit10 of 12 checks passed

The output is a complete, data-backed proposal with a clear recommendation, plan table, migration steps, and testing conditions. A product manager could take it to the CEO with only light edits.

Output

Inkwell pricing and packaging proposal

To: Priya Nair | Proposed launch: January 2027

Recommendation

Do not move everyone to $19. Keep a $15 solo offer and introduce a $25/user agency plan. Launch for new customers in January, subject to validation; migrate existing agencies gradually.

Your 27% calculation is correct only with unchanged paid seats. At $19, losing 21.1% of seats eliminates the uplift. The evidence supports differentiated pricing—not equal increases for customers with very different needs.

What the evidence says

  • Agencies drive revenue. The random account sample contains 80 freelancers, 50 small agencies and 20 larger agencies. Agencies represent 47% of accounts but 90% of sample MRR: $10,605 of $11,805.
  • Freelancer acquisition already struggles. Trial conversion is 9%, versus 31% for small agencies and 37% for larger agencies. Second-month cancellation is 38%, 6% and 0%, respectively. The last figure covers only 22 paying accounts, so is not evidence of zero underlying churn.
  • Price tolerance differs. Median surveyed “getting expensive” prices are $15 for freelancers, $31 for small agencies and $51 for larger agencies. These are directional signals, not measured purchase probabilities or optimal prices.
  • Price is not the only problem. Only 6 of 160 recorded deals show a price objection. Freelancer losses go to Google Docs plus free e-signature; agency losses go to PandaDoc. Sales records may underrepresent self-serve price sensitivity.
  • Workflows justify packaging. Freelancers mainly use one template and e-signature; agencies use the shared content library, approval workflows and CRM sync. Sample quarterly sending ranges also differ: 3–12 proposals for freelancers, 18–63 for small agencies and 95–287 for larger agencies.

Proposed plans

USD, monthly billing; retain the 14-day trial.

PlanPriceIntended customerIncluded
------:------
Solo$15/month; one userFreelancers needing a complete proposal-to-signature workflowTemplate-based proposal creation, sending and e-signature; no shared content library, approval workflows or CRM sync
Team$25/user/month; minimum two usersSmall and larger agencies coordinating proposalsEverything in Solo, plus shared content library, approval workflows and CRM sync

No proposal-volume caps or signature fees at launch. Charge for internal users; recipients/signers are free. One plan applies across an account.

Keep seats as the billing metric: it is familiar and the survey tested per-user prices. Differentiate through collaboration capabilities rather than taxing successful contracts. Signed contract value indicates customer value but is neither collected revenue nor value attributable solely to Inkwell.

Do not launch a third tier yet. Larger agencies show more price headroom, but the files do not establish a distinct premium feature bundle. Validate that opportunity separately rather than inventing enterprise functionality.

Keeping Solo at $15 avoids worsening a fragile funnel; cutting its price is also unsupported. Test activation improvements before assuming price causes freelancer churn.

Revenue implications

Illustrative repricing of the sample, not a company forecast:

ScenarioMonthly revenueChange
------:---:
Current pricing: 787 seats$11,805—
Everyone at $19; unchanged seats$14,953+26.7%
Solo $15; agencies temporarily $19$14,633+24.0%
Solo $15; agencies $25$18,875+59.9%
Final pricing with 20% fewer agency seats$15,340+29.9%

These scenarios assume all sampled agency accounts use Team, freelancers remain, and no other changes. Actual revenue depends on cancellations, seat reductions and plan selection. The final uplift arrives after migration, not in January. Without cost data, this is not a profit forecast.

Existing-customer migration

  1. Give at least 60 days’ notice, with each account’s current bill, transition bill, final bill and dates. If that notice cannot precede January, delay existing-account changes.
  2. Existing freelancers stay at $15, retaining currently included capabilities as legacy entitlements. New Solo customers receive the published bundle. Do not remove capabilities customers already rely on.
  3. Existing agencies move to $19/user at their first eligible renewal, then $25 after six months. Honour any contractual commitments and communicate both steps upfront. Preserve all existing functionality.
  4. Provide account-level outreach for larger agencies and a seat-review tool before renewal. Never interrupt in-flight proposals, signatures or access to signed documents.

Validate before broad rollout

Before January, test $25 Team against the current $15 offer in segment-stratified new-trial cohorts. Measure 60-day retained revenue per trial, conversion, cancellations and seats—not initial conversion alone.

Require higher retained revenue per trial and no more than a proposed three-percentage-point deterioration in agency second-month cancellation before expanding. Extend testing if samples are inconclusive.

Stagger existing-account migration with a temporary unchanged-price comparison group. Monitor retained revenue and seat contraction through both price steps; pause the next cohort if revenue underperforms the comparison group. January should be a controlled launch, not an irreversible announcement.

Grades and run details

Decision model 75 · LLM judge 100
Decision model checks
  • failUses the supplied evidence correctly30%
  • passAddresses the actual decision76%
  • partialRespects explicit constraints27%
  • passIdentifies material uncertainty92%
  • passAvoids unsupported claims28%
  • passProduces the required deliverable9%
  • partialSees who's underpriced100%
  • passAnswers the CEO's flat rise98%
  • passEach tier for a different buyer94%
  • failGets the base of every number right55%
  • passPriced on willingness to pay66%
  • passMoves existing customers fairly98%
Run
Run
#1
Time to output
58 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 6

Uses the supplied evidence correctlyWrongMixedMixed
Gemini 4 Argon · API

The claim that 83% of freelancers find $19 'getting expensive' and 15% 'too expensive' is wrong (actual approx 97% and 10% from survey), misrepresenting the evidence.

GPT-6.1 Sol · API

Every statement about the current situation is supported by the supplied data or arithmetic, with no inventions.

GPT-6 Astra · ChatGPT

Every factual claim about the current situation is drawn directly from the supplied data or follows from it by arithmetic, and nothing is invented.

Identifies material uncertaintyWrongRightRight
Gemini 4 Argon · API

Does not name specific unknowns that could change the decision, bound them, or say how they would be resolved.

GPT-6.1 Sol · API

Names small agency conversion base, inconsistent survey response, testability of prices, and unknown churn/seat contraction; says how to validate.

GPT-6 Astra · ChatGPT

Output names the small sample for agency churn, possible under-representation of self-serve price sensitivity, directional nature of WTP, and missing cost data, and says how testing and comparison groups will resolve them.

Avoids unsupported claimsWrongRightRight
Gemini 4 Argon · API

Presents the incorrect percentages (83% and 15%) as fact without caveat, and makes other unsupported claims like 'near-zero logo churn' without support.

GPT-6.1 Sol · API

Labels survey as stated preferences, acknowledges the limits of loss data, and does not present hypotheses as facts.

GPT-6 Astra · ChatGPT

Interpretations and forecasts are clearly signalled as such ('may underrepresent', 'directional signals', 'not evidence of zero underlying churn'), and confident claims are supported by the evidence.

Produces the required deliverableMixedRightRight
Gemini 4 Argon · API

Delivers a complete proposal with table, within word limit, usable as a brief for the CEO.

GPT-6.1 Sol · API

Contains the required plan table with prices and features, stays under 900 words, and is actionable for the CEO with light edits.

GPT-6 Astra · ChatGPT

The output is a complete pricing and packaging proposal with a table, addressed to Priya Nair, within 900 words, and is usable as-is.

Sees who's underpricedRightWrongRight
Gemini 4 Argon · API

Uses signed value per seat and survey price points to show agencies are heavily underpriced and freelancers are near their limit.

GPT-6.1 Sol · API

Does not use the signed value per seat data to show agencies extract far more value and are underpriced; mentions revenue concentration but omits the large per-seat signed values.

GPT-6 Astra · ChatGPT

Output shows agencies dominate MRR, cites WTP survey medians that put agencies far above $15 and freelancers at $15, and links features to agency workflows, demonstrating they are underpriced while freelancers are near their limit.

Gets the base of every number rightWrongMixedMixed
Gemini 4 Argon · API

The percentages for freelancer 'getting expensive' and 'too expensive' at $19 are incorrectly derived from the survey, indicating calculation errors.

GPT-6.1 Sol · API

All derived percentages and rates (26.7% uplift, 21.1% cancellation threshold, conversion rates) check out against the supplied data and are computed from correct denominators.

GPT-6 Astra · ChatGPT

All computed percentages, revenue scenarios, and the break-even seat loss are derived correctly from the sample data, with bases stated where needed.

All got right 6

Addresses the actual decisionRightRightRight
Gemini 4 Argon · API

Commits early to a three-tier pricing model ($12/$29/$49) and explains the choice clearly for the CEO.

GPT-6.1 Sol · API

Commits to a clear three-tier segmented pricing answer early, addresses the CEO, and says what metric would change the course (non-positive incremental revenue).

GPT-6 Astra · ChatGPT

Output commits early to keeping a $15 solo plan and introducing a $25/user agency plan, and states validation conditions that would change the approach.

Respects explicit constraintsRightRightRight
Gemini 4 Argon · API

Under 900 words, includes the required plan table, and addresses the CEO's request for pricing and packaging.

GPT-6.1 Sol · API

Proposal is within 900 words, includes a plan table, and avoids the prohibited flat $19 for everyone.

GPT-6 Astra · ChatGPT

The proposal is under 900 words, includes a plan table, does not propose a single flat $19 price, and addresses the CEO's request.

Answers the CEO's flat riseRightRightRight
Gemini 4 Argon · API

Explains with data that $19 overcharges freelancers and undercharges agencies, referencing WTP thresholds.

GPT-6.1 Sol · API

Shows $19 exceeds freelancer 'getting expensive' range and falls below agency thresholds, explaining why a flat rise would overcharge one and undercharge the other.

GPT-6 Astra · ChatGPT

It explains that $19 is above freelancers' 'getting expensive' median ($15) and far below agencies' ($31/$51), and quantifies how even a modest seat loss from freelancers would wipe out the revenue gain from a flat rise.

Each tier for a different buyerRightRightRight
Gemini 4 Argon · API

Each tier targets a distinct buyer with features tied to that buyer's usage and needs.

GPT-6.1 Sol · API

Solo, Team, and Business plans are each built for a distinct buyer (freelancer, collaborating small agency, integrated larger agency) with features that buyer values.

GPT-6 Astra · ChatGPT

Solo is built for freelancers with core proposal/e-signature capabilities; Team is built for agencies and includes the collaboration features they value, clearly targeting different buyers.

Priced on willingness to payRightRightRight
Gemini 4 Argon · API

Prices are grounded in the willingness-to-pay data (set below segment 'getting expensive' medians).

GPT-6.1 Sol · API

Prices are set against the survey's 'getting expensive' ranges for each segment and the output notes where evidence is thin (stated preferences, small agency conversion base).

GPT-6 Astra · ChatGPT

The $15 and $25 prices are anchored to the WTP survey medians and the value agencies capture, and the output notes where evidence is thin for a third tier.

Moves existing customers fairlyRightRightRight
Gemini 4 Argon · API

Provides a transition plan: new customers first, freelancers moved to lower price, agencies grandfathered with notice and loyalty bridge.

GPT-6.1 Sol · API

Provides a clear transition plan: 90 days' notice, freelancers stay at $15, agencies move to $19 then to Team/Business, with churn monitoring.

GPT-6 Astra · ChatGPT

Migration plan gives 60 days' notice, keeps freelancers at $15 with legacy entitlements, phases agencies to $19 then $25, ties timing to renewals, and includes outreach and a seat-review tool.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 85% of 68 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Opus 5.5withClaude68.1100.02None
2GPT-6 AstrawithChatGPT70.296.42None
3Sonnet 5.5withAPI69.989.32None
4GPT-6.1 SolwithAPI74.088.72None
5Haiku 5.5withAPI73.776.92None
6GPT-6 LunawithAPI78.464.02None
7Gemini 4 ArgonwithAPI47.659.32None
8Gemini 3.8 FlashwithAPI55.644.22None
9Gemini 3.5 Flash-LitewithGemini41.840.72None

What you get for the cost

GPT-6.1 Sol · API scores within 5 points of Opus 5.5 · Claude at a sixth of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreOpus 5.5 · Claude: task score 85, 122× the cheapestGPT-6 Astra · ChatGPT: task score 84, 108× the cheapestSonnet 5.5 · API: task score 81, 61× the cheapestGPT-6.1 Sol · API: task score 80, 20× the cheapestHaiku 5.5 · API: task score 74, 2.9× the cheapestGPT-6 Luna · API: task score 70, the cheapestGemini 4 Argon · API: task score 52, 15× the cheapestGemini 3.8 Flash · API: task score 48, 9.1× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 40, 2.3× the cheapestOpus 5.5 · ClaudeGPT-6 Astra · ChatGPTSonnet 5.5 · APIGPT-6.1 Sol · APIHaiku 5.5 · APIGPT-6 Luna · APIGemini 4 Argon · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and cost
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API701.0×
Gemini 3.5 Flash-Lite · Gemini402.3×
Haiku 5.5 · API742.9×
Gemini 3.8 Flash · API489.1×
Gemini 4 Argon · API5215×
GPT-6.1 Sol · API8020×
Sonnet 5.5 · API8161×
GPT-6 Astra · ChatGPT84108×
Opus 5.5 · Claude85122×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Setting what you charge, for what, and to whom.

Why it matters

Most products are priced once and never revisited, on a metric that doesn't track value. The fix is rarely a flat rise.

What good looks like

  • Tiers built for distinct buyers
  • A value metric that tracks what customers get
  • Grounded in willingness to pay and cost to serve
  • A plan for existing customers

Deliberately not measured

  • Billing implementation
  • Legal terms
Capability tested

Pricing judgement

The failure we’re looking for

A flat price rise on the wrong value metric

Grading

Decision model and LLM judge, calibrated against a blind PM review