Humanity’s Last PRD
Latest drop: Gemini 4 Argon · 13 Oct
Tasks / Metrics & Experimentation

Experiment specification

Can the model design a test that could actually change the decision?

Task type v1.0 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 28% were usable with at most a quick edit.

Reliably right

  1. Randomises by user100% pass
    Randomises by logged-in user ID across platforms, explicitly citing cross-platform usage and session non-independence.
    Sonnet 5.5 · API · Show the fee sooner
  2. Works around the calendar94% pass
    Addresses both the payments test and the Black Friday freeze, proposes concurrent independent assignment with an interaction check, and avoids a one-week squeeze.
    Sonnet 5.5 · API · Show the fee sooner
  3. Tests one change at a time89% pass
    The free-delivery banner is explicitly excluded from the variant, with an explanation that bundling would prevent attribution, and a separate test is proposed.
    Sonnet 5.5 · API · Showing the delivery fee up front

Where it slips

  1. Says why a test is worth it here42% pass
    The output does not explain why this decision merits a test given the traffic and stakes, nor does it recommend a cheaper alternative.
    Opus 5.5 · Claude · Show the fee sooner
  2. Guardrails with thresholds56% pass
    Guardrail metrics are named but lack explicit numeric thresholds; 'materially worse' and 'drops meaningfully' are too vague to block rollout without further definition.
    Sonnet 5.5 · API · Showing the delivery fee up front
  3. Doesn't let the metric flatter the change61% pass
    The primary metric is from basket entry and explains why last-step completion misleads, but it does not include a basket drop-off guardrail with a threshold to block rollout.
    GPT-6 Astra · ChatGPT · Show the fee sooner

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. An unambiguous primary metric

    Does the spec name one primary metric, explain why an increase is unambiguously good for the user and the business, and say how the result's trustworthiness will be checked?

    Passes when One primary metric with a clear rationale for its direction, plus a planned trust check such as a sample ratio check.

    Ronny KohaviLenny’s Podcast

  2. Decision rule written before the test

    Does the spec say, before the test runs, which result leads to which action (ship, don't ship, iterate), including what happens if the result is inconclusive?

    Passes when Every outcome, including an inconclusive one, maps to a stated action, with the thresholds that separate them.

    Ronny KohaviLenny’s Podcast

  3. Sized from the real traffic

    Is the sample size and duration worked out from the supplied traffic, baseline and smallest effect worth detecting, run in full weekly cycles?

    Passes when Sample size follows from the baseline and the smallest effect worth acting on, the duration follows from the traffic the brief gives (including how units accumulate), and it runs whole weeks.

    Product and growth leadersOn Lenny’s Podcast

  4. Guardrails with thresholds

    Does the spec name the guardrail metrics that would stop a rollout even if the primary metric wins, each with a threshold?

    Passes when The guardrails that matter for this change are named, each with a threshold that would block the rollout.

    Lauryn IsfordLenny’s Podcast

  5. Says why a test is worth it here

    Does the spec weigh whether a test is worth running here: enough traffic to detect an effect that matters, and a decision the result would actually change, and does it say what it would do instead if not?

    Passes when Explains why this decision merits a test given the traffic and stakes, or recommends a cheaper route where they don't.

    Lauryn IsfordLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for checkout at Basketful. We want to test showing the delivery fee in the basket, instead of only at the last step of checkout. Write the experiment spec for Ravi, the growth engineer who'll build it, Ines, the analyst who'll read it out, and Chloe, our Head of Growth, who approves it. Keep it under 700 words. The team's draft plan is below. Fix what needs fixing.

What the model was given5 items: About Basketful, Why we're testing it, Traffic and baselines, Smallest change worth acting on, The team's draft plan
About BasketfulAn online grocery service. Delivery costs £3.99, or is free on orders over £60. Today the fee first appears on the final checkout step.
Why we're testing it12% of support contacts last quarter were about delivery fees the customer hadn't expected. Design believes showing the fee early will build trust; Chloe worries it will put people off before they start checkout.
Traffic and baselinesAbout 40,000 different customers view their basket each week. About half of each week's basket viewers are new that week; the rest came back from earlier weeks. 8% of basket viewers place an order. Of customers who start checkout, 62% complete it. Average order value: £47.
Smallest change worth acting onChloe: “Anything smaller than half a percentage point on orders isn't worth arguing about.”
The team's draft planPrimary metric: checkout completion rate (orders ÷ checkout starts). Split: 50/50 by session. Duration: we'll check the dashboard every day and stop as soon as it's significant. Chloe's addition: “While we're at it, let's put the free-delivery-over-£60 banner in the variant too.”
What a strong answer doesThe answer key the graders mark against

A spec that tests one change and fixes the draft's three problems. The primary metric is orders per basket-viewing customer, not checkout completion rate: showing the fee earlier changes who starts checkout, so completion can rise while orders fall. Customers, not sessions, are randomised, so returning customers see one version. The free-delivery banner stays out of the variant (or gets its own test), or the result can't be attributed. The duration is fixed in advance, with no daily stopping: detecting 0.5pp on an 8% base needs roughly 47,000 customers per arm (about 95,000 in total), and because half of each week's viewers are returning customers, new customers accumulate at about 40,000, 60,000, 80,000 and 100,000 after four weeks, so it runs about four full weeks, not two and a half. Guardrails with thresholds: average order value, fee-related support contacts, and the checkout-start rate as a diagnostic. A sample ratio check before reading results. A decision rule written up front: ship if orders are no worse than 0.5pp down and fee contacts fall; don't ship if orders drop by 0.5pp or more; with a stated plan for an inconclusive result.

Critical failures (cap the score at 40)
  • Uses checkout completion rate (orders ÷ checkout starts) as the primary metric
  • Bundles the free-delivery banner into the variant
Task

Mid-level · v1.4

The outputs

9 setups answered this brief.

Sonnet 5.5API89Usable with a quick editHaiku 5.5API89Usable with a quick editGPT-6.1 SolAPI79Usable with a quick editGPT-6 AstraChatGPT75Needs more workOpus 5.5Claude73Needs more workGPT-6 LunaAPI71Usable with a quick editGemini 4 ArgonAPI59Needs more workGemini 3.8 FlashAPI45Needs more workGemini 3.5 Flash-LiteGemini23Critical failure
Sonnet 5.5API

Usable with a quick edit12 of 13 checks passed

The spec is complete and well-structured; the only small omission is explicit guardrail thresholds, which a PM can add with a quick edit.

Output

Experiment spec: delivery fee in basket

Owner: PM, Checkout | Readers: Ravi (build), Ines (analysis), Chloe (approval)

Hypothesis

Showing the delivery fee in the basket reduces surprise at checkout (12% of last quarter's support contacts) without reducing orders. Chloe's concern is that an early fee deters people before they start checkout, so the test measures that directly.

Changes from the team draft

DraftProblemFix
Primary metric: checkout completionIt only counts people who already started checkout. If the early fee puts people off, they never enter the denominator, so the harm would be hidden.Primary metric is orders per basket viewer.
Split by sessionReturning customers (about half of weekly viewers) would see both versions.Randomise by customer.
Check daily, stop when significantRepeated peeking inflates false positives.Fixed sample size and duration, one readout.
Add the free-delivery banner to the variantWe couldn't tell whether the fee or the banner caused any change.Banner excluded. Test it separately afterwards.

Design

  • Control: fee first shown at final checkout step (today).
  • Variant: basket shows the delivery line: £3.99, or "Free" at £60+. Nothing else changes. The basket and checkout amounts must always match.
  • Unit: customer ID, with a 50/50 split. Assignment is sticky for the whole test. Logged-out users are bucketed by a persistent device ID (Ravi to confirm how many there are).
  • Enrolment: at first basket view during the test. Log an exposure event at that moment.

Metrics

Primary: share of enrolled customers who place at least one order within 7 days of first basket view. Baseline is about 8%.

Guardrails: - Checkout start rate (baseline about 12.9%, i.e. 8% ÷ 62%). - Revenue per enrolled customer (baseline about £3.76, i.e. 8% × £47). - Delivery-fee support contacts per 1,000 customers. This is the benefit metric, but volumes are low, so read it directionally.

Diagnostics (not decision metrics): - Checkout completion (orders ÷ checkout starts). Read it only alongside the primary metric, since it's conditional on starts. - New vs returning customers, and baskets under vs over £60. Both are exploratory.

Sample size and duration

  • Smallest effect worth detecting: 0.5 percentage points on orders (8.0% → 8.5%), per Chloe.
  • Requirement: 80% power, 5% two-sided significance. This needs about 46,000 customers per arm, roughly 92,000 in total.
  • Time to reach it: Week 1 brings about 40,000 unique customers. After that, only about 20,000 new customers enrol per week, because returning customers are already assigned. That reaches about 100,000 by the end of week 4.
  • Duration: enrol for 4 full weeks, plus 7 days of follow-up for the last cohort. Readout is at about 5 weeks, with no early stopping. Ines can monitor data quality but not results.
  • Limit: the test can't reliably detect effects smaller than 0.5pp. A "no significant difference" result means "no effect larger than about 0.5pp", not "no effect".

Data quality checks (before readout)

  • Sample ratio check: arms should be within normal variation of 50/50.
  • Exposure is logged for every enrolled customer.
  • Fee shown in basket equals fee charged, in all cases.

Decision rule (Chloe to confirm)

  • Ship if the 95% confidence interval for the change in orders rules out a drop of 0.5pp or more, no guardrail is materially worse, and delivery-fee contacts do not rise.
  • Don't ship if orders fall by 0.5pp or more, or revenue per customer drops meaningfully.
  • Otherwise, treat as inconclusive and discuss. Do not extend the test to "find" significance.

Next

If the variant ships or is neutral, run a follow-up test of the free-delivery-over-£60 banner, where the fee is already visible. Its effect is likely concentrated in baskets just under £60.

Open items

  • Ravi: logged-out identity handling; a flag to switch the variant off quickly.
  • Chloe: confirm the decision rule and that the 0.5pp threshold means absolute percentage points on orders per basket viewer.
  • Ines: confirm the 7-day order attribution and the support-contact tagging for fee-related contacts.

Grades and run details

Decision model 85 · LLM judge 93
Decision model checks
  • passUses the supplied evidence correctly23%
  • passAddresses the actual decision56%
  • partialRespects explicit constraints20%
  • passIdentifies material uncertainty82%
  • partialAvoids unsupported claims22%
  • passProduces the required deliverable52%
  • passTests one change at a time100%
  • passFixes the duration up front97%
  • passAn unambiguous primary metric88%
  • passDecision rule written before the test95%
  • passSized from the real traffic28%
  • partialGuardrails with thresholds54%
  • partialSays why a test is worth it here86%
Run
Run
#1
API response time
29 s
Submitted
30 Sept 2026
Gemini 4 ArgonAPI

Needs more than a quick edit6 of 13 checks passed

The spec is strong but needs a sample ratio check, explicit guardrail thresholds, and a rule for the guardrail‐decline outcome before a PM can use it without rework.

Output

Experiment Spec: Upfront Delivery Fee in Basket

To: Ravi (Eng), Ines (Analytics), Chloe (Head of Growth) From: Checkout PM Subject: Experiment Spec – Showing Delivery Fee in Basket

1. Background & Hypothesis

Last quarter, 12% of support contacts were complaints about unexpected delivery fees. Currently, our £3.99 fee (free on orders >£60) is hidden until the final checkout step.

  • Hypothesis: Showing the delivery fee in the basket will build trust and reduce support contacts without hurting overall conversion, rather than scaring users away early.

2. Scope & Variants (Fixing the Draft)

Chloe suggested adding a "Free delivery over £60" banner to the variant. Recommendation: Hold this for a follow-up test. If we test the fee and the promotional banner together, we won’t know if a change in conversion came from fee transparency (friction) or the £60 threshold incentive (upsell). However, because our delivery fee is dynamic, the variant line item must accurately reflect the £60 rule so users aren't misled.

  • Control: Current basket experience (no delivery fee shown until final checkout step).
  • Variant: Add a single, standard-text line item to the basket summary:
  • If basket < £60: “Estimated Delivery: £3.99 (Free over £60)”
  • If basket ≥ £60: “Estimated Delivery: FREE”
  • (Note for Chloe: If we want to test a high-visibility promotional banner/progress bar to drive AOV, let's fast-follow with that as V2).

3. Metrics (Fixing the Draft)

The draft plan used Checkout Completion Rate (Orders ÷ Checkout Starts) as the primary metric. We must change this. Moving the fee to the basket will likely cause sticker-shock drop-offs to happen in the basket rather than at the end of checkout. This will shrink the number of "Checkout Starts" and artificially inflate Checkout Completion Rate, even if total orders drop.

Because the intervention happens in the basket, our denominator must be basket viewers:

  • Primary Metric: Basket-to-Order Conversion Rate (Unique customers placing an order ÷ Unique customers viewing the basket). Baseline: 8.0%.
  • Secondary Metrics:
  • Average Order Value (AOV): To see if the subtle text nudges baskets over £60. Baseline: £47.
  • Basket-to-Checkout Start Rate & Checkout Completion Rate: To observe the shift in where drop-off occurs.
  • Support Contacts (Delivery Fee): Tracked via contact form tags/UTMs where possible.
  • Guardrail Metric: Net Revenue per Basket Viewer (ensuring total revenue doesn't drop even if conversion holds).

4. Randomization & Targeting (Fixing the Draft)

The draft plan splits traffic 50/50 by session. We must change this to a 50/50 split by Customer/User ID (or persistent device cookie for logged-out users). Groceries are a high-consideration purchase; 50% of our weekly traffic consists of returning users. Splitting by session means a single customer could see the fee in the morning, and not see it in the evening, causing confusion and polluting our data.

  • Trigger: User lands on `/basket`.

5. Sample Size & Duration (Fixing the Draft)

The draft plan suggests checking daily and stopping when significant. We cannot do this ("peeking" skyrockets false-positive rates). We need a fixed horizon.

  • Minimum Detectable Effect (MDE): Chloe set our threshold at 0.5 percentage points (an absolute shift from 8.0% to 8.5% or 7.5%).
  • Sample Size Needed: At an 8% baseline, 80% power, and 95% significance (two-tailed), detecting a 0.5pp change requires ~46,200 unique users per variant (~92,400 total).
  • Duration: 4 Weeks (Fixed).
  • Traffic math: We get 40,000 unique basket viewers in Week 1, and ~20,000 net-new viewers each subsequent week (since 50% are repeat visitors). We will hit ~100,000 unique users across the test at the end of Week 4. Running in full weekly increments also accounts for weekend vs. weekday shopping cycles.

6. Decision Criteria

After 4 weeks, Ines will run the readout (segmented by New vs. Returning users): * Ship Variant: If Basket-to-Order CR is flat (within ±0.5pp) or significantly positive, AND AOV/Revenue guardrails do not significantly decline. * Rollback: If Basket-to-Order CR drops by ≥ 0.5pp.

Sign-offs Needed: * Chloe: Approve scope (holding the promo banner for V2) and 4-week runtime. * Ines: Confirm sample size math and user-level tracking. * Ravi: Confirm engineering effort for user-level bucketing on `/basket`.

Claims the judge couldn’t find in the brief

  • Groceries are a high-consideration purchase.

Grades and run details

Decision model 81 · LLM judge 43
Decision model checks
  • passUses the supplied evidence correctly1%
  • partialAddresses the actual decision73%
  • passRespects explicit constraints23%
  • passIdentifies material uncertainty48%
  • partialAvoids unsupported claims23%
  • passProduces the required deliverable64%
  • passTests one change at a time100%
  • passFixes the duration up front98%
  • partialAn unambiguous primary metric72%
  • passDecision rule written before the test33%
  • passSized from the real traffic58%
  • partialGuardrails with thresholds88%
  • partialSays why a test is worth it here95%
Run
Run
#1
API response time
13 s
Submitted
13 Oct 2026
GPT-6 AstraChatGPT

Needs more than a quick edit11 of 13 checks passed

Guardrail thresholds are missing and the spec defers their definition, so a PM would need to add them before the spec is actionable.

Output

Experiment: show delivery fees in the basket

Owner/build: Ravi · Analysis: Ines · Approval: Chloe Decision: Should Basketful show delivery charges before checkout?

Hypothesis and scope

Showing the fee in the basket will reduce fee surprises, but could discourage ordering. Unexpected delivery fees accounted for 12% of support contacts last quarter; this is motivation, not a baseline customer complaint rate.

Test fee visibility only. Do not add the proposed free-delivery promotional banner: it introduces a second change and would prevent us attributing results to earlier disclosure.

Experience

  • Control: Current experience; delivery fee first appears at the final checkout step.
  • Treatment: Show a clearly labelled delivery line beside the basket total: £3.99 delivery, or Free delivery when eligible. Update the fee and total as the basket changes.
  • Use the existing pricing service and eligibility rules. Confirm behaviour at exactly £60 before launch; “over £60” must not silently become “£60 or more.”
  • Leave checkout, pricing and other messaging unchanged.

Eligibility and assignment

Include customers viewing a non-empty, orderable basket. Exclude staff, bots and test accounts using rules fixed before launch.

Randomise 50/50 by customer, not session, at their first eligible basket view. Persist assignment across visits and devices where identity is available. For signed-out customers, use a stable first-party identifier; Ravi must document identity merging and cross-device limitations before launch.

Analyse everyone assigned, whether or not they start checkout or successfully see the fee: intention to treat.

Metrics

Primary: Customer order conversion — percentage of assigned customers placing at least one order within seven days of assignment. Report treatment minus control in percentage points, with a two-sided 95% confidence interval.

Checkout completion is diagnostic only: treatment may change who starts checkout, so orders ÷ checkout starts could improve while overall ordering falls.

Secondary/diagnostic: - Checkout-start rate and checkout completion. - Delivery-fee-related support contacts per assigned customer within seven days, using consistent tagging. - Orders and revenue per assigned customer; average order value and free-delivery eligibility as diagnostics.

Guardrails: Contribution margin per assigned customer, if reliably available; otherwise net revenue per assigned customer as an explicitly limited proxy. Also monitor basket/checkout errors. Chloe and Ines must approve numerical economic and reliability tolerances before launch.

Sample size and schedule

Chloe’s smallest worthwhile order-conversion change is 0.5 percentage points: 8.0% to 8.5%, not a 0.5% relative lift.

At 5% two-sided significance and 80% power, detecting that increase requires approximately 48,000 customers per arm, or 96,000 total.

Ines must first verify that the 8% baseline matches the eligibility rules and seven-day outcome window; recalculate and freeze the sample target if necessary.

Do not count returning customers again. Expected recruitment is approximately:

  • Week 1: 40,000 unique customers.
  • Each subsequent week: 20,000 additional customers.
  • Four weeks: approximately 100,000 unique customers.

Enroll for at least four complete weeks and until the frozen sample target is reached, ending at a full-week boundary. Then wait seven days for the final cohort’s outcomes: roughly five weeks to readout.

Monitoring and decision

Ravi validates assignment persistence, fee accuracy, exposure/order linkage and sample-ratio balance before ramp-up.

Daily checks are for instrumentation and predefined safety breaches—not efficacy stopping. No “stop when significant,” sample extensions based on results, or repeated significance decisions.

Ines delivers one final primary analysis. Chloe’s default rollout rule is: - Estimated conversion gain at least +0.5 points; - 95% confidence interval excludes zero; - Guardrails pass.

Otherwise, do not roll out on this result alone. A non-significant result does not prove no harm. Secondary trust improvements inform follow-up, not a post-hoc replacement for the primary success rule.

Grades and run details

Decision model 77 · LLM judge 79
Decision model checks
  • passUses the supplied evidence correctly18%
  • partialAddresses the actual decision35%
  • partialRespects explicit constraints20%
  • passIdentifies material uncertainty72%
  • partialAvoids unsupported claims27%
  • passProduces the required deliverable27%
  • passTests one change at a time97%
  • partialFixes the duration up front88%
  • passAn unambiguous primary metric55%
  • passDecision rule written before the test95%
  • passSized from the real traffic54%
  • partialGuardrails with thresholds98%
  • partialSays why a test is worth it here94%
Run
Run
#1
Time to output
47 s
Submitted
1 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 6

Uses the supplied evidence correctlyRightMixedRight
Sonnet 5.5 · API

All statements about the current situation are taken directly from the brief or derived by correct arithmetic.

Gemini 4 Argon · API

The output claims that 'groceries are a high-consideration purchase' without support from the brief or supplied context.

GPT-6 Astra · ChatGPT

All factual claims about the current situation are taken from the brief or follow by arithmetic; the pricing service is a reasonable inference from the delivery fee rules.

Identifies material uncertaintyRightMixedRight
Sonnet 5.5 · API

It names specific unknowns (logged-out identity, decision rule confirmation, attribution window) and says how to resolve them.

Gemini 4 Argon · API

It does not name specific unknowns that could change the decision (e.g., unmeasurable trust gains) or explain how an inconclusive result (e.g., guardrail decline while conversion holds) would be resolved.

GPT-6 Astra · ChatGPT

Names specific unknowns (baseline verification, guardrail tolerances, identity merging limitations) and says how they will be resolved before launch.

Avoids unsupported claimsRightWrongRight
Sonnet 5.5 · API

Hypotheses and forecasts are clearly labelled as such, and no unsupported claims are presented as fact.

Gemini 4 Argon · API

The statement 'Groceries are a high-consideration purchase' is presented as fact without evidence from the supplied material.

GPT-6 Astra · ChatGPT

Hypotheses are labelled as such, and confident claims are supported by the brief.

An unambiguous primary metricRightWrongRight
Sonnet 5.5 · API

The primary metric is orders per enrolled customer, with a clear rationale, and a sample ratio check is planned.

Gemini 4 Argon · API

The spec does not include a planned trust check (e.g., sample ratio check) to verify randomization integrity before reading results.

GPT-6 Astra · ChatGPT

Names one primary metric (customer order conversion), explains why an increase is unambiguously good, and includes a sample-ratio balance check.

Decision rule written before the testRightMixedRight
Sonnet 5.5 · API

Ship, don't ship, and inconclusive outcomes are all mapped to actions with thresholds before the test runs.

Gemini 4 Argon · API

No action is stated for the outcome where AOV/Revenue guardrails decline but Basket-to-Order CR does not drop by ≥0.5pp, leaving that inconclusive result unmapped.

GPT-6 Astra · ChatGPT

States a pre-defined rule: ship if conversion gain ≥0.5pp, CI excludes zero, guardrails pass; otherwise do not roll out, including for inconclusive results.

Says why a test is worth it hereRightWrongWrong
Sonnet 5.5 · API

The spec demonstrates the test is feasible by sizing it to detect the smallest effect that matters within the available traffic, implicitly justifying the test.

Gemini 4 Argon · API

The spec does not discuss whether the test is worth running given the traffic and stakes, nor what the alternative would be.

GPT-6 Astra · ChatGPT

Does not explain why a test is worth running given the traffic and stakes; it assumes the test is justified without weighing alternatives.

All got wrong 1

Guardrails with thresholdsWrongWrongWrong
Sonnet 5.5 · API

Guardrail metrics are named but lack explicit numeric thresholds; 'materially worse' and 'drops meaningfully' are too vague to block rollout without further definition.

Gemini 4 Argon · API

Guardrails are named but no numerical thresholds are set to define when they would block rollout.

GPT-6 Astra · ChatGPT

Guardrail metrics are named but no numerical thresholds are set; the spec defers approval of tolerances, leaving the spec incomplete.

All got right 6

Addresses the actual decisionRightRightRight
Sonnet 5.5 · API

The spec commits to a clear decision rule (ship, don't ship, inconclusive) with thresholds, framed for Chloe and the team.

Gemini 4 Argon · API

The output commits to a clear experiment design and decision rule (ship/rollback) early, and states what conditions would change the call.

GPT-6 Astra · ChatGPT

Commits to a clear answer (run the experiment, roll out only if conversion gain ≥0.5pp, CI excludes zero, guardrails pass) and states what would change it.

Respects explicit constraintsRightRightRight
Sonnet 5.5 · API

The output is an experiment spec under 700 words, addresses Ravi, Ines, and Chloe, and fixes the draft's problems as requested.

Gemini 4 Argon · API

The spec is under 700 words, addressed to the three named readers, and fixes the draft's issues as requested.

GPT-6 Astra · ChatGPT

The spec is under 700 words, addresses the named readers, and is in the requested experiment-spec form.

Produces the required deliverableRightRightRight
Sonnet 5.5 · API

The spec is complete, in the right form, for the named readers, and within the word limit.

Gemini 4 Argon · API

The required experiment spec is present, complete, and usable by the named readers.

GPT-6 Astra · ChatGPT

The experiment spec is complete, under 700 words, and usable by Ravi, Ines, and Chloe with light edits.

Tests one change at a timeRightRightRight
Sonnet 5.5 · API

The free-delivery banner is explicitly excluded from the variant, with an explanation that bundling would prevent attribution, and a separate test is proposed.

Gemini 4 Argon · API

It explicitly removes the free-delivery banner from the variant, explains why bundling it would make the result uninterpretable, and offers a follow-up test.

GPT-6 Astra · ChatGPT

Explicitly excludes the free-delivery banner from the variant and explains that adding it would prevent attribution.

Fixes the duration up frontRightRightRight
Sonnet 5.5 · API

The spec sets a fixed 4-week enrolment plus 7-day follow-up, explains that daily peeking inflates false positives, and forbids early stopping.

Gemini 4 Argon · API

It replaces daily stopping with a fixed 4-week duration and explains that peeking inflates false positives.

GPT-6 Astra · ChatGPT

Sets a fixed duration of at least four full weeks, states daily checks are for safety only, and prohibits stopping on significance.

Sized from the real trafficRightRightRight
Sonnet 5.5 · API

Sample size is calculated from the 8% baseline and 0.5pp effect, and duration is derived from the 40k weekly traffic and new-customer accumulation, running whole weeks.

Gemini 4 Argon · API

Sample size (≈46k per arm) is derived from the 0.5pp MDE and 8% baseline, and the 4-week duration follows from the traffic accumulation arithmetic (40k + 20k/week).

GPT-6 Astra · ChatGPT

Sample size (48k/arm) is calculated from the 8% baseline and 0.5pp MDE; duration uses the traffic data (40k unique/week, half new) and runs whole weeks.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 87% of 39 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Sonnet 5.5API85.296.42None
2Haiku 5.5API83.196.72None
3Opus 5.5Claude83.279.02None
4GPT-6 AstraChatGPT81.376.02None
5GPT-6 LunaAPI83.465.22None
6GPT-6.1 SolAPI77.769.32None
7Gemini 4 ArgonAPI79.758.12None
8Gemini 3.8 FlashAPI69.947.92None
9Gemini 3.5 Flash-LiteGemini55.413.321 capped

What you get for the cost

Haiku 5.5 · API scores within 2 points of Sonnet 5.5 · API at about a tenth of the cost.

Task score against relative cost for each setup. The table below has the same figures.20304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreSonnet 5.5 · API: task score 91, 33× the cheapestHaiku 5.5 · API: task score 89, 2.7× the cheapestOpus 5.5 · Claude: task score 78, 49× the cheapestGPT-6 Astra · ChatGPT: task score 76, 79× the cheapestGPT-6 Luna · API: task score 73, the cheapestGPT-6.1 Sol · API: task score 73, 13× the cheapestGemini 4 Argon · API: task score 66, 11× the cheapestGemini 3.8 Flash · API: task score 56, 7.4× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 32, 2.1× the cheapest, 1 critical failureSonnet 5.5 · APIHaiku 5.5 · APIOpus 5.5 · ClaudeGPT-6 Astra · ChatGPTGPT-6 Luna · APIGPT-6.1 Sol · APIGemini 4 Argon · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costAt least one critical failure

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Specifying an A/B test before running it.

Why it matters

Most tests are underpowered, or measure a metric nobody agrees is good.

What good looks like

  • One primary metric everyone agrees on the direction of
  • Decision rule stated up front
  • Guardrails
  • Power considered

Deliberately not measured

    Capability tested

    Test design

    The failure we’re looking for

    A test with no decision attached

    Grading

    Decision model and LLM judge, calibrated against a blind PM review