Humanity’s Last PRD
Latest drop: Gemini 4 Argon · 13 Oct
Tasks / Metrics & Experimentation

Experiment specification

Can the model design a test that could actually change the decision?

Task type v1.0 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 28% were usable with at most a quick edit.

Reliably right

  1. Randomises by user100% pass
    Randomises by logged-in user ID across platforms, explicitly citing cross-platform usage and session non-independence.
    Sonnet 5.5 · API · Show the fee sooner
  2. Works around the calendar94% pass
    Addresses both the payments test and the Black Friday freeze, proposes concurrent independent assignment with an interaction check, and avoids a one-week squeeze.
    Sonnet 5.5 · API · Show the fee sooner
  3. Tests one change at a time89% pass
    The free-delivery banner is explicitly excluded from the variant, with an explanation that bundling would prevent attribution, and a separate test is proposed.
    Sonnet 5.5 · API · Showing the delivery fee up front

Where it slips

  1. Says why a test is worth it here42% pass
    The output does not explain why this decision merits a test given the traffic and stakes, nor does it recommend a cheaper alternative.
    Opus 5.5 · Claude · Show the fee sooner
  2. Guardrails with thresholds56% pass
    Guardrail metrics are named but lack explicit numeric thresholds; 'materially worse' and 'drops meaningfully' are too vague to block rollout without further definition.
    Sonnet 5.5 · API · Showing the delivery fee up front
  3. Doesn't let the metric flatter the change61% pass
    The primary metric is from basket entry and explains why last-step completion misleads, but it does not include a basket drop-off guardrail with a threshold to block rollout.
    GPT-6 Astra · ChatGPT · Show the fee sooner

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. An unambiguous primary metric

    Does the spec name one primary metric, explain why an increase is unambiguously good for the user and the business, and say how the result's trustworthiness will be checked?

    Passes when One primary metric with a clear rationale for its direction, plus a planned trust check such as a sample ratio check.

    Ronny KohaviLenny’s Podcast

  2. Decision rule written before the test

    Does the spec say, before the test runs, which result leads to which action (ship, don't ship, iterate), including what happens if the result is inconclusive?

    Passes when Every outcome, including an inconclusive one, maps to a stated action, with the thresholds that separate them.

    Ronny KohaviLenny’s Podcast

  3. Sized from the real traffic

    Is the sample size and duration worked out from the supplied traffic, baseline and smallest effect worth detecting, run in full weekly cycles?

    Passes when Sample size follows from the baseline and the smallest effect worth acting on, the duration follows from the traffic the brief gives (including how units accumulate), and it runs whole weeks.

    Product and growth leadersOn Lenny’s Podcast

  4. Guardrails with thresholds

    Does the spec name the guardrail metrics that would stop a rollout even if the primary metric wins, each with a threshold?

    Passes when The guardrails that matter for this change are named, each with a threshold that would block the rollout.

    Lauryn IsfordLenny’s Podcast

  5. Says why a test is worth it here

    Does the spec weigh whether a test is worth running here: enough traffic to detect an effect that matters, and a decision the result would actually change, and does it say what it would do instead if not?

    Passes when Explains why this decision merits a test given the traffic and stakes, or recommends a cheaper route where they don't.

    Lauryn IsfordLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for checkout at Basketful. We want to test showing the delivery fee in the basket, instead of only at the last step of checkout. Write the experiment spec for Ravi, the growth engineer who'll build it, Ines, the analyst who'll read it out, and Chloe, our Head of Growth, who approves it. Keep it under 700 words. The team's draft plan is below. Fix what needs fixing.

What the model was given5 items: About Basketful, Why we're testing it, Traffic and baselines, Smallest change worth acting on, The team's draft plan
About BasketfulAn online grocery service. Delivery costs £3.99, or is free on orders over £60. Today the fee first appears on the final checkout step.
Why we're testing it12% of support contacts last quarter were about delivery fees the customer hadn't expected. Design believes showing the fee early will build trust; Chloe worries it will put people off before they start checkout.
Traffic and baselinesAbout 40,000 different customers view their basket each week. About half of each week's basket viewers are new that week; the rest came back from earlier weeks. 8% of basket viewers place an order. Of customers who start checkout, 62% complete it. Average order value: £47.
Smallest change worth acting onChloe: “Anything smaller than half a percentage point on orders isn't worth arguing about.”
The team's draft planPrimary metric: checkout completion rate (orders ÷ checkout starts). Split: 50/50 by session. Duration: we'll check the dashboard every day and stop as soon as it's significant. Chloe's addition: “While we're at it, let's put the free-delivery-over-£60 banner in the variant too.”
What a strong answer doesThe answer key the graders mark against

A spec that tests one change and fixes the draft's three problems. The primary metric is orders per basket-viewing customer, not checkout completion rate: showing the fee earlier changes who starts checkout, so completion can rise while orders fall. Customers, not sessions, are randomised, so returning customers see one version. The free-delivery banner stays out of the variant (or gets its own test), or the result can't be attributed. The duration is fixed in advance, with no daily stopping: detecting 0.5pp on an 8% base needs roughly 47,000 customers per arm (about 95,000 in total), and because half of each week's viewers are returning customers, new customers accumulate at about 40,000, 60,000, 80,000 and 100,000 after four weeks, so it runs about four full weeks, not two and a half. Guardrails with thresholds: average order value, fee-related support contacts, and the checkout-start rate as a diagnostic. A sample ratio check before reading results. A decision rule written up front: ship if orders are no worse than 0.5pp down and fee contacts fall; don't ship if orders drop by 0.5pp or more; with a stated plan for an inconclusive result.

Critical failures (cap the score at 40)
  • Uses checkout completion rate (orders ÷ checkout starts) as the primary metric
  • Bundles the free-delivery banner into the variant
Task

Mid-level · v1.4

The outputs

9 setups answered this brief.

Sonnet 5.5API89Usable with a quick editHaiku 5.5API89Usable with a quick editGPT-6.1 SolAPI79Usable with a quick editGPT-6 AstraChatGPT75Needs more workOpus 5.5Claude73Needs more workGPT-6 LunaAPI71Usable with a quick editGemini 4 ArgonAPI59Needs more workGemini 3.8 FlashAPI45Needs more workGemini 3.5 Flash-LiteGemini23Critical failure
GPT-6 LunaAPI

Usable with a quick edit7 of 13 checks passed

The core spec is solid and fixes the draft's main problems; a PM could add the missing explanation for no peeking, a sample ratio check, guardrail thresholds, and an approval recommendation with quick edits.

Output

Basket delivery-fee experiment

For: Ravi (build), Ines (analysis), Chloe (approval) Decision requested: Approve a test of showing the existing delivery fee in the basket.

Hypothesis and scope

Showing the delivery cost before checkout will reduce surprise and fee-related support contacts, without materially reducing orders.

  • Control: Current experience; delivery fee first appears on the final checkout step.
  • Variant: Show the applicable fee in the basket: “Delivery £3.99” below the free-delivery threshold, or “Free delivery” when the basket qualifies. Keep the existing “free on orders over £60” rule and wording.
  • Change nothing else. Do not add the free-delivery banner: that would test a second intervention and make the result hard to interpret.

Build and assignment — Ravi

Randomise 50/50 by customer, not session, and keep each customer in the same arm across visits and devices where identity is available. Stratify assignment by new versus returning basket viewers. Exclude staff and test accounts. Log assignment, basket value/fee shown, basket views, checkout starts, orders, order value, and relevant support contacts.

Measurement — Ines

Primary metric: Basket-viewer conversion: percentage of assigned customers who place at least one order within 7 days of their first eligible basket view. Count each customer once, in their assigned arm (intention-to-treat). This captures any deterrence before checkout; checkout completion alone would miss it.

Guardrails / secondary measures: Checkout starts per basket viewer; checkout completion (orders ÷ checkout starts); average order value and revenue per assigned customer; delivery-fee-related support contacts per assigned customer. Support contacts may be too sparse for a definitive test, so treat them as directional, not a substitute for the primary outcome.

Sample, duration, and analysis

Baseline basket-viewer conversion is 8%. Power the test to detect a 0.5 percentage-point absolute difference (Chloe’s smallest worthwhile order effect), with 80% power and a two-sided 5% significance level. This requires approximately 95,000 unique customers total (about 47,500 per arm), assuming independent customer-level outcomes.

There are about 40,000 basket viewers weekly, but many return across weeks. Run for at least four full weeks and until the sample target is reached; then allow 7 days for outcomes to mature before reading results. If the target is not reached, extend in full weeks. Ines should confirm the final sample-size calculation and report actual enrolment.

Do not stop early for significance or check daily results to make a stopping decision. Monitor only for implementation or customer-harm issues. Analyse the primary metric at the customer level, with a 95% confidence interval and the pre-specified two-sided test.

Decision

Recommend rollout if the variant improves basket-viewer conversion by at least 0.5 percentage points, the result is statistically significant, and guardrails show no material harm. Do not roll out if there is significant harm. If results are inconclusive, report that clearly; do not interpret “not significant” as proof of no effect.

Grades and run details

Decision model 85 · LLM judge 57
Decision model checks
  • passUses the supplied evidence correctly33%
  • passAddresses the actual decision30%
  • passRespects explicit constraints18%
  • passIdentifies material uncertainty37%
  • passAvoids unsupported claims31%
  • passProduces the required deliverable46%
  • passTests one change at a time94%
  • partialFixes the duration up front97%
  • partialAn unambiguous primary metric76%
  • passDecision rule written before the test30%
  • passSized from the real traffic31%
  • partialGuardrails with thresholds86%
  • partialSays why a test is worth it here91%
Run
Run
#1
API response time
22 s
Submitted
30 Sept 2026
Gemini 3.8 FlashAPI

Needs more than a quick edit4 of 13 checks passed

The duration is miscalculated (3 weeks instead of 4), guardrail thresholds are missing, and the decision rule lacks an inconclusive branch. These require rework beyond a quick edit.

Output

Experiment Spec: Delivery Fee Transparency in Basket

To: Chloe (Approval), Ravi (Build), Ines (Analysis) From: Checkout PM

---

1. Objective & Hypothesis

  • Context: 12% of support contacts complain about surprise delivery fees at final checkout.
  • Hypothesis: Displaying the delivery fee (£3.99, or "Free" if >£60) upfront in the basket will set accurate expectations, reduce support contacts, and build customer trust without reducing net orders.
  • Risk (Chloe’s concern): Upfront cost shock could cause drop-off before checkout begins.

---

2. Corrections to the Draft Plan

1. Primary Metric Changed: Draft used Checkout Completion Rate (Orders ÷ Starts). Correction: Use Basket-to-Order Conversion Rate (Orders ÷ Basket Viewers). Measuring only checkout starts introduces severe selection bias—if fee visibility deters low-intent users at the basket, checkout completion would artificially increase while overall orders plummet. 2. Unit of Randomization Changed: Draft used session-level. Correction: Randomize by User ID (falling back to persistent anonymous visitor ID). 50% of weekly viewers are repeat shoppers; session-level randomization would expose returning users to conflicting experiences. 3. No Peeking / Fixed Duration: Draft suggested stopping once "significant." Correction: Continuous monitoring with early stopping inflates false-positive rates to >30%. We will fix sample size upfront and run for complete weekly cycles. 4. Scope Control (Chloe’s Banner): Cut the promotional free-delivery banner from this test. Adding a promotional banner introduces a confounding variable, making it impossible to isolate whether fee transparency helped or hurt. We will test the banner in a follow-up experiment.

---

3. Variants (For Ravi)

  • Allocation: 50/50 split at User/Persistent Device ID level. Triggered on basket view.
  • Control (A): Current basket. No delivery fee shown (fee appears on final checkout step as today).
  • Variant (B): Add a delivery fee line item to the Basket Order Summary:
  • Basket < £60: `Delivery: £3.99`
  • Basket ≥ £60: `Delivery: FREE`
  • Include delivery fee in the displayed Basket Total.
  • Note: Do not add promotional progress bars or marketing banners.

---

4. Metrics Framework (For Ines)

  • Primary Metric: Basket-to-Order Conversion Rate (Total Unique Ordering Users ÷ Total Unique Basket Viewers).
  • Guardrail Metrics:
  • Average Order Value (AOV baseline: £47).
  • Basket-to-Checkout Start Rate (tracks where abandonment occurs).
  • Total Net Revenue (Orders × AOV).
  • Secondary / Diagnostic Metrics:
  • Delivery-fee-related customer support ticket rate per 1,000 orders.
  • Checkout Completion Rate (Checkout Starts → Orders).

---

5. Sample Size & Run Time (For Ines & Chloe)

  • Baseline Conversion: 8.0% of basket viewers place an order.
  • Minimum Detectable Effect (MDE): Absolute ±0.5 percentage points (7.5% or 8.5%, a 6.25% relative change), per Chloe's threshold.
  • Statistical Parameters: $\alpha = 0.05$ (two-tailed), Power ($1 - \beta$) = 80%.
  • Required Sample: ~47,000 unique users per variant (~94,000 total unique basket viewers).
  • Duration: 3 full weeks (21 days).
  • With 40,000 viewers/week (~20k new, ~20k repeat), 3 weeks yields ~80,000–90,000 unique basket viewers across three full grocery-shopping cycles.
  • Ines will run the read-out strictly after the 21-day window closes.

---

6. Decision & Rollout Criteria (For Chloe)

  • Ship Variant: Primary conversion does not drop by more than Chloe’s threshold of -0.5pp (non-inferiority) AND delivery-related support contacts decrease significantly.
  • Iterate / Re-test: Basket-to-order conversion drops by >0.5pp. In this scenario, early fee disclosure hurts conversion, justifying Chloe’s concern; we then test the £60 threshold banner/progress bar to offset drop-off.
  • Kill Immediately: Overall revenue drops >5% statistically significantly at the end of Week 1 (safety check only; not an early win stop).

Grades and run details

Decision model 58 · LLM judge 36
Decision model checks
  • failUses the supplied evidence correctly33%
  • partialAddresses the actual decision64%
  • partialRespects explicit constraints22%
  • passIdentifies material uncertainty62%
  • partialAvoids unsupported claims20%
  • passProduces the required deliverable30%
  • passTests one change at a time100%
  • partialFixes the duration up front31%
  • partialAn unambiguous primary metric73%
  • partialDecision rule written before the test27%
  • partialSized from the real traffic95%
  • partialGuardrails with thresholds80%
  • partialSays why a test is worth it here85%
Run
Run
#1
API response time
19 s
Submitted
1 Oct 2026
Haiku 5.5API

Usable with a quick edit12 of 13 checks passed

The spec is complete, directly addresses all team members, and a PM could use it as-is, only needing to fill in the two baselines that are clearly flagged.

Output

Experiment Spec: Delivery Fee in Basket

To: Ravi (build), Ines (analysis), Chloe (approval) From: PM, Checkout Status: For approval

What changed from the draft

  • Primary metric moved from checkout completion to basket-to-order conversion. Completion only measures people who reach checkout, so it would miss the risk Chloe raised: people leaving before checkout.
  • Randomise by customer, not session. Half of each week's viewers are returning customers. Session splits would show the same person both versions.
  • Fixed duration, pre-set sample size. Checking daily and stopping at first significance inflates false positives.
  • Free-delivery banner removed. It changes the same thing we're measuring, so we couldn't attribute any effect. We'll propose it as a separate test.

Hypothesis

Showing the delivery fee in the basket changes basket-to-order conversion. We expect an increase of at least 0.5 percentage points (8.0% to 8.5%). The test is two-sided, so a decrease would also be recorded and acted on.

Variants

  • Control: Basket as today. Fee appears only on the final checkout step.
  • Variant: Basket shows a delivery line under the subtotal: "Delivery: £3.99" or "Delivery: free" once the basket is over £60. Checkout is unchanged.

Ravi: The line must update live as the basket crosses the £60 threshold, and must match the fee charged at checkout exactly.

Population and randomisation

  • All customers who view their basket during the test period.
  • Randomise 50/50 by customer ID at first basket view in the test. Assignment stays fixed for the whole test.
  • Run a sample ratio mismatch (SRM) check on day 1. If the split deviates from 50/50 at p < 0.01, pause and investigate before continuing.

Metrics

Primary: Basket-to-order conversion. Customers who place an order within 7 days of their first basket view in the test, divided by customers who viewed their basket. Baseline: 8%.

Guardrails: - Checkout start rate (checkout starts ÷ basket viewers). Ines to pull baseline. - Checkout completion rate. Baseline: 62%. - Average order value. Baseline: £47. - Delivery-fee support contacts per 1,000 orders. Ines to pull baseline; currently 12% of all contacts.

Sample size and duration

  • Baseline 8.0%, minimum effect 0.5 percentage points absolute, two-sided α = 0.05, power 80%.
  • Requires about 47,500 customers per arm (95,000 total).
  • Traffic is about 40,000 basket viewers per week, of whom about 20,000 are new. Unique customers therefore accumulate at roughly 40k in week 1 and 20k per week after that.
  • Plan: 4 weeks of enrolment, then a 7-day conversion window, for a fixed end date of [date]. Ines to confirm the unique-customer count, since the 20k-new-per-week figure is an estimate.
  • No decisions before the end date, with one exception: at week 2, Ines runs a harm-only check on guardrails. We stop early only if a guardrail shows harm at p < 0.001.

Decision rules

Ship if: - The primary metric is significantly positive (p < 0.05), with an estimated uplift of at least 0.5 percentage points, and - No guardrail is significantly worse.

Don't ship if: - The primary result is not significant, or - It is significant but below 0.5 points (not worth acting on, per Chloe), or - A guardrail is significantly worse.

We don't iterate on a null result by re-slicing the data until something looks significant.

Analysis (Ines)

  • Analyse at the customer level, intention-to-treat: all assigned customers, regardless of whether they saw the fee line.
  • Report absolute difference in percentage points and relative difference, each with a 95% confidence interval.
  • Report all guardrails.
  • Pre-registered segment: new vs returning customers. Returning customers may already know the fee, so the effect may differ. Treat any other segment cuts as exploratory.

Timeline and sign-off

  • Build complete: [date]
  • Launch: [date]
  • Fixed end: [date]
  • Readout: [date + 3 working days]

Approved by: _____ Chloe, Head of Growth Date: ___

Grades and run details

Decision model 77 · LLM judge 100
Decision model checks
  • failUses the supplied evidence correctly19%
  • partialAddresses the actual decision23%
  • passRespects explicit constraints14%
  • passIdentifies material uncertainty48%
  • partialAvoids unsupported claims20%
  • passProduces the required deliverable45%
  • passTests one change at a time98%
  • passFixes the duration up front82%
  • passAn unambiguous primary metric67%
  • passDecision rule written before the test89%
  • partialSized from the real traffic49%
  • passGuardrails with thresholds62%
  • partialSays why a test is worth it here89%
Run
Run
#1
API response time
29 s
Submitted
8 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 10

Uses the supplied evidence correctlyRightMixedMixed
GPT-6 Luna · API

All statements about the current situation are taken directly from the brief or supplied context.

Gemini 3.8 Flash · API

All statements about the current situation are taken directly from the brief and are accurate.

Haiku 5.5 · API

All statements about the current situation are directly from the supplied context or simple arithmetic; no facts are invented.

Addresses the actual decisionMixedRightRight
GPT-6 Luna · API

The output does not commit to a clear answer on whether to approve the test; it only provides a rollout decision rule for after the test.

Gemini 3.8 Flash · API

The spec commits to a clear decision rule with ship, iterate, and kill actions based on results.

Haiku 5.5 · API

The output commits to a clear experimental design and includes decision rules that map results to specific actions (ship/don't ship), which addresses the team's need to decide how to test and what to act on.

Identifies material uncertaintyRightMixedRight
GPT-6 Luna · API

It acknowledges that support contacts may be too sparse and that inconclusive results are possible, and says how to handle them.

Gemini 3.8 Flash · API

No plan is given for an inconclusive result (e.g., if the difference is not significant or support contacts don't decrease), and the sample-size uncertainty from the 3-week duration is not addressed.

Haiku 5.5 · API

The output names uncertainty in the new-customer accumulation estimate and asks Ines to confirm, and notes the segment effect may differ; it bounds where possible.

Avoids unsupported claimsRightWrongRight
GPT-6 Luna · API

No interpretations or forecasts are presented as established facts; the sparsity note is hedged as 'may'.

Gemini 3.8 Flash · API

The claim that continuous monitoring inflates false-positive rates to >30% is presented as fact without support from the brief.

Haiku 5.5 · API

Hypotheses, forecasts, and reasoning are clearly labelled as such; no confident claims go beyond the supplied evidence.

Fixes the duration up frontWrongWrongRight
GPT-6 Luna · API

It sets a fixed duration but does not explain why stopping at the first significant day inflates false positives.

Gemini 3.8 Flash · API

The duration is set at 3 weeks, but the required sample of ~94,000 unique users needs about 4 weeks to accumulate; the fixed duration is not correctly derived from the sample size.

Haiku 5.5 · API

The spec sets a fixed 4-week enrolment plus 7-day conversion window, explains that daily peeking inflates false positives, and only allows a harm-based early stop on guardrails.

An unambiguous primary metricWrongWrongRight
GPT-6 Luna · API

No sample ratio check or other trust check is planned, only a confirmation of sample size and enrolment.

Gemini 3.8 Flash · API

The primary metric is clearly stated and rationalized, but no trust check (e.g., sample ratio check) is planned before reading the results.

Haiku 5.5 · API

A single primary metric (basket-to-order conversion) is named with a clear rationale for its direction, and a sample ratio mismatch check is planned.

Decision rule written before the testMixedWrongRight
GPT-6 Luna · API

The inconclusive outcome maps only to 'report clearly', not to a decision on whether to ship, iterate, or not ship.

Gemini 3.8 Flash · API

The rule does not cover an inconclusive outcome (e.g., non-significant result or support contacts not decreasing) and lacks a stated action for that case.

Haiku 5.5 · API

Every possible result is mapped to an action: ship if significant with ≥0.5pp uplift and guardrails safe; don't ship otherwise; no iteration on null results.

Sized from the real trafficRightWrongRight
GPT-6 Luna · API

Sample size is correctly derived from the 8% baseline and 0.5pp effect, and duration accounts for weekly traffic and returning customers.

Gemini 3.8 Flash · API

The sample size calculation is correct, but the 3-week duration yields only ~80,000–90,000 unique viewers, which is below the required ~94,000; the duration is not sufficient.

Haiku 5.5 · API

Sample size is correctly derived from the 8% baseline, 0.5pp effect, 80% power; duration follows from the 40k weekly traffic and half-new accumulation, running whole weeks.

Guardrails with thresholdsWrongWrongRight
GPT-6 Luna · API

Guardrail metrics are listed but no thresholds are given that would block a rollout.

Gemini 3.8 Flash · API

Guardrail metrics like AOV and basket-to-checkout start rate are listed but no explicit thresholds are set that would block a rollout.

Haiku 5.5 · API

Checkout start rate, completion rate, average order value, and delivery-fee support contacts are listed as guardrails, with a requirement that they are not significantly worse for rollout to proceed.

Says why a test is worth it hereWrongWrongRight
GPT-6 Luna · API

The output does not explain why a test is worth running here given the traffic and stakes.

Gemini 3.8 Flash · API

The spec does not weigh whether a test is worth running given the traffic and stakes, nor does it suggest an alternative if a test is not justified.

Haiku 5.5 · API

The spec demonstrates the test is powered for the smallest change that matters, and the decision rules show the result would change the team's action, which justifies running the test.

All got right 3

Respects explicit constraintsRightRightRight
GPT-6 Luna · API

The spec is under 700 words, addresses Ravi, Ines, and Chloe, and fixes the draft's problems.

Gemini 3.8 Flash · API

The output is under 700 words, addresses the three named readers, and fixes the draft's problems.

Haiku 5.5 · API

The spec is within the 700-word limit, is addressed to Ravi, Ines, and Chloe, and proposes an actionable test plan.

Produces the required deliverableRightRightRight
GPT-6 Luna · API

The experiment spec is complete, under 700 words, and usable by the named readers.

Gemini 3.8 Flash · API

The output is a complete experiment spec in the requested format, usable by the named readers with light edits.

Haiku 5.5 · API

The output is a complete experiment spec with sections for build, analysis, and sign-off, and is usable by the intended readers with minimal editing.

Tests one change at a timeRightRightRight
GPT-6 Luna · API

The spec explicitly keeps the free-delivery banner out of the variant and explains why bundling it would make the result uninterpretable.

Gemini 3.8 Flash · API

The free-delivery banner is explicitly removed from the variant and a follow-up test is proposed, with a clear explanation of the confounding risk.

Haiku 5.5 · API

The free-delivery banner is explicitly removed from the variant with a clear explanation that it would confound attribution, and it is suggested as a separate test.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 87% of 39 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Sonnet 5.5API85.296.42None
2Haiku 5.5API83.196.72None
3Opus 5.5Claude83.279.02None
4GPT-6 AstraChatGPT81.376.02None
5GPT-6 LunaAPI83.465.22None
6GPT-6.1 SolAPI77.769.32None
7Gemini 4 ArgonAPI79.758.12None
8Gemini 3.8 FlashAPI69.947.92None
9Gemini 3.5 Flash-LiteGemini55.413.321 capped

What you get for the cost

Haiku 5.5 · API scores within 2 points of Sonnet 5.5 · API at about a tenth of the cost.

Task score against relative cost for each setup. The table below has the same figures.20304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreSonnet 5.5 · API: task score 91, 33× the cheapestHaiku 5.5 · API: task score 89, 2.7× the cheapestOpus 5.5 · Claude: task score 78, 49× the cheapestGPT-6 Astra · ChatGPT: task score 76, 79× the cheapestGPT-6 Luna · API: task score 73, the cheapestGPT-6.1 Sol · API: task score 73, 13× the cheapestGemini 4 Argon · API: task score 66, 11× the cheapestGemini 3.8 Flash · API: task score 56, 7.4× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 32, 2.1× the cheapest, 1 critical failureSonnet 5.5 · APIHaiku 5.5 · APIOpus 5.5 · ClaudeGPT-6 Astra · ChatGPTGPT-6 Luna · APIGPT-6.1 Sol · APIGemini 4 Argon · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costAt least one critical failure

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Specifying an A/B test before running it.

Why it matters

Most tests are underpowered, or measure a metric nobody agrees is good.

What good looks like

  • One primary metric everyone agrees on the direction of
  • Decision rule stated up front
  • Guardrails
  • Power considered

Deliberately not measured

    Capability tested

    Test design

    The failure we’re looking for

    A test with no decision attached

    Grading

    Decision model and LLM judge, calibrated against a blind PM review