Humanity’s Last PRD
Latest drop: Haiku 5.5 · 8 Oct
Tasks / Metrics & Experimentation

Experiment specification

Can the model design a test that could actually change the decision?

Task type v1.0 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 31% were usable with at most a quick edit.

Reliably right

  1. Randomises by user100% pass
    Randomises by logged-in user ID across platforms, explicitly citing cross-platform usage and session non-independence.
    Sonnet 5.5 · API · Show the fee sooner
  2. Works around the calendar94% pass
    Addresses both the payments test and the Black Friday freeze, proposes concurrent independent assignment with an interaction check, and avoids a one-week squeeze.
    Sonnet 5.5 · API · Show the fee sooner
  3. Addresses the actual decision91% pass
    Commits to a clear experiment plan with a decision rule that maps every outcome (ship, don't ship, kill, extend) to an action.
    Sonnet 5.5 · API · Show the fee sooner

Where it slips

  1. Guardrails with thresholds50% pass
    Guardrail metrics are named but lack explicit numeric thresholds; 'materially worse' and 'drops meaningfully' are too vague to block rollout without further definition.
    Sonnet 5.5 · API · Showing the delivery fee up front
  2. Says why a test is worth it here59% pass
    The output does not explain why this decision merits a test given the traffic and stakes, nor does it recommend a cheaper alternative.
    Opus 5.5 · Claude · Show the fee sooner
  3. Uses the supplied evidence correctly64% pass
    Unsupported claims about the traffic data's adequacy and the code freeze's preapproval requirement are presented as facts, not in the brief.
    GPT-6.1 Sol · API · Show the fee sooner

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. An unambiguous primary metric

    Does the spec name one primary metric, explain why an increase is unambiguously good for the user and the business, and say how the result's trustworthiness will be checked?

    Passes when One primary metric with a clear rationale for its direction, plus a planned trust check such as a sample ratio check.

    Ronny KohaviLenny’s Podcast

  2. Decision rule written before the test

    Does the spec say, before the test runs, which result leads to which action (ship, don't ship, iterate), including what happens if the result is inconclusive?

    Passes when Every outcome, including an inconclusive one, maps to a stated action, with the thresholds that separate them.

    Ronny KohaviLenny’s Podcast

  3. Sized from the real traffic

    Is the sample size and duration worked out from the supplied traffic, baseline and smallest effect worth detecting, run in full weekly cycles?

    Passes when Sample size follows from the baseline and the smallest effect worth acting on, the duration follows from the traffic the brief gives (including how units accumulate), and it runs whole weeks.

    Product and growth leadersOn Lenny’s Podcast

  4. Guardrails with thresholds

    Does the spec name the guardrail metrics that would stop a rollout even if the primary metric wins, each with a threshold?

    Passes when The guardrails that matter for this change are named, each with a threshold that would block the rollout.

    Lauryn IsfordLenny’s Podcast

  5. Says why a test is worth it here

    Does the spec weigh whether a test is worth running here: enough traffic to detect an effect that matters, and a decision the result would actually change, and does it say what it would do instead if not?

    Passes when Explains why this decision merits a test given the traffic and stakes, or recommends a cheaper route where they don't.

    Lauryn IsfordLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for checkout at Basketful. We want to test showing the delivery fee in the basket, instead of only at the last step of checkout. Write the experiment spec for Ravi, the growth engineer who'll build it, Ines, the analyst who'll read it out, and Chloe, our Head of Growth, who approves it. Keep it under 700 words. The team's draft plan is below. Fix what needs fixing.

What the model was given5 items: About Basketful, Why we're testing it, Traffic and baselines, Smallest change worth acting on, The team's draft plan
About BasketfulAn online grocery service. Delivery costs £3.99, or is free on orders over £60. Today the fee first appears on the final checkout step.
Why we're testing it12% of support contacts last quarter were about delivery fees the customer hadn't expected. Design believes showing the fee early will build trust; Chloe worries it will put people off before they start checkout.
Traffic and baselinesAbout 40,000 different customers view their basket each week. About half of each week's basket viewers are new that week; the rest came back from earlier weeks. 8% of basket viewers place an order. Of customers who start checkout, 62% complete it. Average order value: £47.
Smallest change worth acting onChloe: “Anything smaller than half a percentage point on orders isn't worth arguing about.”
The team's draft planPrimary metric: checkout completion rate (orders ÷ checkout starts). Split: 50/50 by session. Duration: we'll check the dashboard every day and stop as soon as it's significant. Chloe's addition: “While we're at it, let's put the free-delivery-over-£60 banner in the variant too.”
What a strong answer doesThe answer key the graders mark against

A spec that tests one change and fixes the draft's three problems. The primary metric is orders per basket-viewing customer, not checkout completion rate: showing the fee earlier changes who starts checkout, so completion can rise while orders fall. Customers, not sessions, are randomised, so returning customers see one version. The free-delivery banner stays out of the variant (or gets its own test), or the result can't be attributed. The duration is fixed in advance, with no daily stopping: detecting 0.5pp on an 8% base needs roughly 47,000 customers per arm (about 95,000 in total), and because half of each week's viewers are returning customers, new customers accumulate at about 40,000, 60,000, 80,000 and 100,000 after four weeks, so it runs about four full weeks, not two and a half. Guardrails with thresholds: average order value, fee-related support contacts, and the checkout-start rate as a diagnostic. A sample ratio check before reading results. A decision rule written up front: ship if orders are no worse than 0.5pp down and fee contacts fall; don't ship if orders drop by 0.5pp or more; with a stated plan for an inconclusive result.

Critical failures (cap the score at 40)
  • Uses checkout completion rate (orders ÷ checkout starts) as the primary metric
  • Bundles the free-delivery banner into the variant
Task

Mid-level · v1.3

The outputs

8 setups answered this brief.

Sonnet 5.5API93Usable with a quick editHaiku 5.5API93Usable with a quick editGPT-6 AstraChatGPT80Needs more workGPT-6.1 SolAPI79Usable with a quick editGPT-6 LunaAPI77Usable with a quick editOpus 5.5Claude75Needs more workGemini 3.8 FlashAPI48Needs more workGemini 3.5 Flash-LiteGemini18Critical failure
Haiku 5.5API

Usable with a quick edit12 of 13 checks passed

The spec is complete, directly addresses all team members, and a PM could use it as-is, only needing to fill in the two baselines that are clearly flagged.

Output

Experiment Spec: Delivery Fee in Basket

To: Ravi (build), Ines (analysis), Chloe (approval) From: PM, Checkout Status: For approval

What changed from the draft

  • Primary metric moved from checkout completion to basket-to-order conversion. Completion only measures people who reach checkout, so it would miss the risk Chloe raised: people leaving before checkout.
  • Randomise by customer, not session. Half of each week's viewers are returning customers. Session splits would show the same person both versions.
  • Fixed duration, pre-set sample size. Checking daily and stopping at first significance inflates false positives.
  • Free-delivery banner removed. It changes the same thing we're measuring, so we couldn't attribute any effect. We'll propose it as a separate test.

Hypothesis

Showing the delivery fee in the basket changes basket-to-order conversion. We expect an increase of at least 0.5 percentage points (8.0% to 8.5%). The test is two-sided, so a decrease would also be recorded and acted on.

Variants

  • Control: Basket as today. Fee appears only on the final checkout step.
  • Variant: Basket shows a delivery line under the subtotal: "Delivery: £3.99" or "Delivery: free" once the basket is over £60. Checkout is unchanged.

Ravi: The line must update live as the basket crosses the £60 threshold, and must match the fee charged at checkout exactly.

Population and randomisation

  • All customers who view their basket during the test period.
  • Randomise 50/50 by customer ID at first basket view in the test. Assignment stays fixed for the whole test.
  • Run a sample ratio mismatch (SRM) check on day 1. If the split deviates from 50/50 at p < 0.01, pause and investigate before continuing.

Metrics

Primary: Basket-to-order conversion. Customers who place an order within 7 days of their first basket view in the test, divided by customers who viewed their basket. Baseline: 8%.

Guardrails: - Checkout start rate (checkout starts ÷ basket viewers). Ines to pull baseline. - Checkout completion rate. Baseline: 62%. - Average order value. Baseline: £47. - Delivery-fee support contacts per 1,000 orders. Ines to pull baseline; currently 12% of all contacts.

Sample size and duration

  • Baseline 8.0%, minimum effect 0.5 percentage points absolute, two-sided α = 0.05, power 80%.
  • Requires about 47,500 customers per arm (95,000 total).
  • Traffic is about 40,000 basket viewers per week, of whom about 20,000 are new. Unique customers therefore accumulate at roughly 40k in week 1 and 20k per week after that.
  • Plan: 4 weeks of enrolment, then a 7-day conversion window, for a fixed end date of [date]. Ines to confirm the unique-customer count, since the 20k-new-per-week figure is an estimate.
  • No decisions before the end date, with one exception: at week 2, Ines runs a harm-only check on guardrails. We stop early only if a guardrail shows harm at p < 0.001.

Decision rules

Ship if: - The primary metric is significantly positive (p < 0.05), with an estimated uplift of at least 0.5 percentage points, and - No guardrail is significantly worse.

Don't ship if: - The primary result is not significant, or - It is significant but below 0.5 points (not worth acting on, per Chloe), or - A guardrail is significantly worse.

We don't iterate on a null result by re-slicing the data until something looks significant.

Analysis (Ines)

  • Analyse at the customer level, intention-to-treat: all assigned customers, regardless of whether they saw the fee line.
  • Report absolute difference in percentage points and relative difference, each with a 95% confidence interval.
  • Report all guardrails.
  • Pre-registered segment: new vs returning customers. Returning customers may already know the fee, so the effect may differ. Treat any other segment cuts as exploratory.

Timeline and sign-off

  • Build complete: [date]
  • Launch: [date]
  • Fixed end: [date]
  • Readout: [date + 3 working days]

Approved by: _____ Chloe, Head of Growth Date: ___

Grades and run details

Decision model 85 · LLM judge 100
Decision model checks
  • failUses the supplied evidence correctly33%
  • passAddresses the actual decision79%
  • passRespects explicit constraints14%
  • passIdentifies material uncertainty45%
  • passAvoids unsupported claims28%
  • passProduces the required deliverable34%
  • passTests one change at a time95%
  • passFixes the duration up front97%
  • passAn unambiguous primary metric77%
  • passDecision rule written before the test41%
  • partialSized from the real traffic40%
  • partialGuardrails with thresholds91%
  • passSays why a test is worth it here62%
Run
Run
#1
API response time
29 s
Submitted
8 Oct 2026
Opus 5.5Claude

Needs more than a quick edit9 of 13 checks passed

The spec exceeds the 700-word limit and lacks explicit guardrail thresholds for the final ship decision, so it needs more than a quick wording edit.

Output

Experiment Spec: Delivery Fee Shown in Basket

Owner: Checkout PM · Build: Ravi · Analysis: Ines · Approver: Chloe

Why we're testing this

Today the £3.99 delivery fee (free over £60) first appears on the final checkout step. Last quarter, 12% of support contacts were about unexpected delivery fees. Design expects that showing the fee earlier will build trust. Chloe's concern is that it will put people off before they start checkout. This test tells us which effect is bigger.

Hypothesis

Showing the delivery fee in the basket will change the share of basket viewers who place an order. We don't assume a direction. The test is two-sided because either outcome is plausible.

Variants

  • Control: Current experience. The fee appears only at the final checkout step.
  • Variant: The basket shows a "Delivery: £3.99" line, or "Free" when the basket is over £60. The same fee still appears at checkout.

Change from draft: The free-delivery-over-£60 banner is removed from this test. If both changes ship together, we can't tell which one caused any result. The banner could also raise orders while the fee line lowers them, so the two effects would cancel and we'd learn nothing. We'll test the banner as a follow-up once this result is in.

Primary metric

Order conversion: the share of customers who view their basket during the test and go on to place at least one order.

Change from draft: The draft used checkout completion (orders ÷ checkout starts). That metric would be misleading here. If the variant puts off fee-sensitive customers before checkout, the people who remain in checkout are the ones already willing to pay. Completion would then rise even if total orders fell. Order conversion counts everyone who saw the basket, so it catches losses at either stage.

Secondary and guardrail metrics

  • Checkout start rate and checkout completion rate. These are diagnostic only and show where any change happens.
  • Average order value. Seeing the fee may push some customers over £60.
  • Revenue per basket viewer. This includes delivery fee income.
  • Fee-related support contacts per 1,000 orders. This is the problem we set out to fix.

Randomisation

  • Unit: customer (logged-in account ID). The draft split by session, which needs to change. About half of each week's basket viewers are returning customers. With a session split, they would see the fee in the basket on one visit and not on the next. That would contaminate both groups and confuse customers.
  • Split: 50/50, sticky for the whole test.
  • Ravi: Please log the assignment when a customer first views their basket, and confirm that logged-out traffic is either excluded or bucketed consistently.

Sample size and duration

  • Assumptions: baseline 8% order conversion, two-sided test at α = 0.05, 80% power.
  • Minimum effect we want to detect: 0.5 percentage points (8.0% → 8.5%). We're reading Chloe's threshold as absolute points on order conversion. That is roughly a 6% relative change.
  • Customers needed: about 47,500 per arm, or about 95,000 total.

How long that takes: About 40,000 customers view their basket in week 1, then about 20,000 new customers each week after. That gives roughly 40k, 60k, 80k and 100k cumulative customers. The test therefore runs for 4 full weeks.

Caveat for Ines: The 8% baseline is per basket viewer per week. Over a 4-week window, each customer has more chances to order, so the per-customer baseline will be higher. At 12%, for example, we'd need about 135,000 customers, which means about 6 weeks. Before launch, Ines will calculate the 4-week per-customer baseline from historical data. The final duration will be fixed from that number and recorded here.

Stopping rule

Change from draft: We won't stop the test as soon as the dashboard shows significance. If we check every day and stop on the first significant result, the false-positive rate climbs well above 5%. The dashboard will often show early "wins" that aren't real.

  • The test runs for the pre-set duration, in full weeks only.
  • The dashboard can be watched for bugs and sample-ratio mismatch. Nobody makes ship decisions from it mid-test.
  • Only reason to stop early: a guardrail failure. That means a broken experience, a sample-ratio mismatch (p < 0.001), or order conversion down by more than 1.5 points.

Decision rules

The confidence interval (CI) below is the interval around the difference in order conversion between the variant and control.

Result on order conversionDecision
Up by a significant amountShip the variant
Down by a significant amountDon't ship. Look into other ways to reduce the support contacts
Not significant, and the CI excludes −0.5 pointsShip. We lose no meaningful number of orders and gain trust and fewer support contacts
Not significant, and the CI includes −0.5 pointsTreat as inconclusive. Chloe decides whether to extend the test or drop the change

Sign-off

  • ☐ Ravi: build plus assignment logging
  • ☐ Ines: baseline check and final duration
  • ☐ Chloe: approval

Grades and run details

Decision model 85 · LLM judge 71
Decision model checks
  • failUses the supplied evidence correctly10%
  • passAddresses the actual decision75%
  • partialRespects explicit constraints18%
  • passIdentifies material uncertainty90%
  • passAvoids unsupported claims46%
  • passProduces the required deliverable55%
  • passTests one change at a time100%
  • passFixes the duration up front97%
  • passAn unambiguous primary metric34%
  • passDecision rule written before the test97%
  • passSized from the real traffic57%
  • partialGuardrails with thresholds88%
  • passSays why a test is worth it here91%
Run
Run
#1
Time to output
31 s
Submitted
30 Sept 2026
Sonnet 5.5API

Usable with a quick edit12 of 13 checks passed

The spec is complete and well-structured; the only small omission is explicit guardrail thresholds, which a PM can add with a quick edit.

Output

Experiment spec: delivery fee in basket

Owner: PM, Checkout | Readers: Ravi (build), Ines (analysis), Chloe (approval)

Hypothesis

Showing the delivery fee in the basket reduces surprise at checkout (12% of last quarter's support contacts) without reducing orders. Chloe's concern is that an early fee deters people before they start checkout, so the test measures that directly.

Changes from the team draft

DraftProblemFix
Primary metric: checkout completionIt only counts people who already started checkout. If the early fee puts people off, they never enter the denominator, so the harm would be hidden.Primary metric is orders per basket viewer.
Split by sessionReturning customers (about half of weekly viewers) would see both versions.Randomise by customer.
Check daily, stop when significantRepeated peeking inflates false positives.Fixed sample size and duration, one readout.
Add the free-delivery banner to the variantWe couldn't tell whether the fee or the banner caused any change.Banner excluded. Test it separately afterwards.

Design

  • Control: fee first shown at final checkout step (today).
  • Variant: basket shows the delivery line: £3.99, or "Free" at £60+. Nothing else changes. The basket and checkout amounts must always match.
  • Unit: customer ID, with a 50/50 split. Assignment is sticky for the whole test. Logged-out users are bucketed by a persistent device ID (Ravi to confirm how many there are).
  • Enrolment: at first basket view during the test. Log an exposure event at that moment.

Metrics

Primary: share of enrolled customers who place at least one order within 7 days of first basket view. Baseline is about 8%.

Guardrails: - Checkout start rate (baseline about 12.9%, i.e. 8% ÷ 62%). - Revenue per enrolled customer (baseline about £3.76, i.e. 8% × £47). - Delivery-fee support contacts per 1,000 customers. This is the benefit metric, but volumes are low, so read it directionally.

Diagnostics (not decision metrics): - Checkout completion (orders ÷ checkout starts). Read it only alongside the primary metric, since it's conditional on starts. - New vs returning customers, and baskets under vs over £60. Both are exploratory.

Sample size and duration

  • Smallest effect worth detecting: 0.5 percentage points on orders (8.0% → 8.5%), per Chloe.
  • Requirement: 80% power, 5% two-sided significance. This needs about 46,000 customers per arm, roughly 92,000 in total.
  • Time to reach it: Week 1 brings about 40,000 unique customers. After that, only about 20,000 new customers enrol per week, because returning customers are already assigned. That reaches about 100,000 by the end of week 4.
  • Duration: enrol for 4 full weeks, plus 7 days of follow-up for the last cohort. Readout is at about 5 weeks, with no early stopping. Ines can monitor data quality but not results.
  • Limit: the test can't reliably detect effects smaller than 0.5pp. A "no significant difference" result means "no effect larger than about 0.5pp", not "no effect".

Data quality checks (before readout)

  • Sample ratio check: arms should be within normal variation of 50/50.
  • Exposure is logged for every enrolled customer.
  • Fee shown in basket equals fee charged, in all cases.

Decision rule (Chloe to confirm)

  • Ship if the 95% confidence interval for the change in orders rules out a drop of 0.5pp or more, no guardrail is materially worse, and delivery-fee contacts do not rise.
  • Don't ship if orders fall by 0.5pp or more, or revenue per customer drops meaningfully.
  • Otherwise, treat as inconclusive and discuss. Do not extend the test to "find" significance.

Next

If the variant ships or is neutral, run a follow-up test of the free-delivery-over-£60 banner, where the fee is already visible. Its effect is likely concentrated in baskets just under £60.

Open items

  • Ravi: logged-out identity handling; a flag to switch the variant off quickly.
  • Chloe: confirm the decision rule and that the 0.5pp threshold means absolute percentage points on orders per basket viewer.
  • Ines: confirm the 7-day order attribution and the support-contact tagging for fee-related contacts.

Grades and run details

Decision model 92 · LLM judge 93
Decision model checks
  • passUses the supplied evidence correctly7%
  • passAddresses the actual decision63%
  • partialRespects explicit constraints21%
  • passIdentifies material uncertainty83%
  • passAvoids unsupported claims43%
  • passProduces the required deliverable75%
  • passTests one change at a time100%
  • passFixes the duration up front100%
  • passAn unambiguous primary metric95%
  • passDecision rule written before the test94%
  • passSized from the real traffic29%
  • partialGuardrails with thresholds93%
  • passSays why a test is worth it here89%
Run
Run
#1
API response time
29 s
Submitted
30 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 4

Uses the supplied evidence correctlyMixedMixedRight
Haiku 5.5 · API

All statements about the current situation are directly from the supplied context or simple arithmetic; no facts are invented.

Opus 5.5 · Claude

All factual claims about the current situation and the draft plan are taken directly from the brief or supplied context, with no inventions.

Sonnet 5.5 · API

All statements about the current situation are taken directly from the brief or derived by correct arithmetic.

Respects explicit constraintsRightWrongRight
Haiku 5.5 · API

The spec is within the 700-word limit, is addressed to Ravi, Ines, and Chloe, and proposes an actionable test plan.

Opus 5.5 · Claude

The output is approximately 771 words, exceeding the 700-word limit stated in the brief.

Sonnet 5.5 · API

The output is an experiment spec under 700 words, addresses Ravi, Ines, and Chloe, and fixes the draft's problems as requested.

Produces the required deliverableRightMixedRight
Haiku 5.5 · API

The output is a complete experiment spec with sections for build, analysis, and sign-off, and is usable by the intended readers with minimal editing.

Opus 5.5 · Claude

The spec is the right form and addresses the named readers, but it exceeds the 700-word limit, so it is not within the requested length.

Sonnet 5.5 · API

The spec is complete, in the right form, for the named readers, and within the word limit.

Guardrails with thresholdsRightWrongWrong
Haiku 5.5 · API

Checkout start rate, completion rate, average order value, and delivery-fee support contacts are listed as guardrails, with a requirement that they are not significantly worse for rollout to proceed.

Opus 5.5 · Claude

Guardrail metrics are listed but no thresholds are set for them to block a rollout; the only threshold given (order conversion down >1.5pp) is for early stopping, not the final ship decision.

Sonnet 5.5 · API

Guardrail metrics are named but lack explicit numeric thresholds; 'materially worse' and 'drops meaningfully' are too vague to block rollout without further definition.

All got right 9

Addresses the actual decisionRightRightRight
Haiku 5.5 · API

The output commits to a clear experimental design and includes decision rules that map results to specific actions (ship/don't ship), which addresses the team's need to decide how to test and what to act on.

Opus 5.5 · Claude

The spec commits to a clear decision rule mapping every test outcome (including inconclusive) to a ship/don't-ship/iterate action, framed for Chloe as approver.

Sonnet 5.5 · API

The spec commits to a clear decision rule (ship, don't ship, inconclusive) with thresholds, framed for Chloe and the team.

Identifies material uncertaintyRightRightRight
Haiku 5.5 · API

The output names uncertainty in the new-customer accumulation estimate and asks Ines to confirm, and notes the segment effect may differ; it bounds where possible.

Opus 5.5 · Claude

It identifies the uncertainty around the per-customer baseline over the test window and delegates resolution to Ines, and the decision rule handles the inconclusive case.

Sonnet 5.5 · API

It names specific unknowns (logged-out identity, decision rule confirmation, attribution window) and says how to resolve them.

Avoids unsupported claimsRightRightRight
Haiku 5.5 · API

Hypotheses, forecasts, and reasoning are clearly labelled as such; no confident claims go beyond the supplied evidence.

Opus 5.5 · Claude

Hypotheses and concerns are attributed to Design and Chloe, and the explanation for not bundling the banner is presented as a plausible scenario, not established fact.

Sonnet 5.5 · API

Hypotheses and forecasts are clearly labelled as such, and no unsupported claims are presented as fact.

Tests one change at a timeRightRightRight
Haiku 5.5 · API

The free-delivery banner is explicitly removed from the variant with a clear explanation that it would confound attribution, and it is suggested as a separate test.

Opus 5.5 · Claude

The free-delivery banner is explicitly removed from the variant, with a clear explanation that bundling would make attribution impossible, and a follow-up test is proposed.

Sonnet 5.5 · API

The free-delivery banner is explicitly excluded from the variant, with an explanation that bundling would prevent attribution, and a separate test is proposed.

Fixes the duration up frontRightRightRight
Haiku 5.5 · API

The spec sets a fixed 4-week enrolment plus 7-day conversion window, explains that daily peeking inflates false positives, and only allows a harm-based early stop on guardrails.

Opus 5.5 · Claude

It replaces daily stopping with a fixed 4-week duration and explains that peeking inflates false positives; early stopping is allowed only for guardrail failures.

Sonnet 5.5 · API

The spec sets a fixed 4-week enrolment plus 7-day follow-up, explains that daily peeking inflates false positives, and forbids early stopping.

An unambiguous primary metricRightRightRight
Haiku 5.5 · API

A single primary metric (basket-to-order conversion) is named with a clear rationale for its direction, and a sample ratio mismatch check is planned.

Opus 5.5 · Claude

Order conversion (basket viewers who place an order) is the single primary metric, its direction is clearly good for the business, and a sample-ratio mismatch check is planned.

Sonnet 5.5 · API

The primary metric is orders per enrolled customer, with a clear rationale, and a sample ratio check is planned.

Decision rule written before the testRightRightRight
Haiku 5.5 · API

Every possible result is mapped to an action: ship if significant with ≥0.5pp uplift and guardrails safe; don't ship otherwise; no iteration on null results.

Opus 5.5 · Claude

The decision table covers significant up, significant down, non-significant with CI above -0.5pp, and inconclusive, each with a stated action.

Sonnet 5.5 · API

Ship, don't ship, and inconclusive outcomes are all mapped to actions with thresholds before the test runs.

Sized from the real trafficRightRightRight
Haiku 5.5 · API

Sample size is correctly derived from the 8% baseline, 0.5pp effect, 80% power; duration follows from the 40k weekly traffic and half-new accumulation, running whole weeks.

Opus 5.5 · Claude

Sample size is calculated from the 8% baseline and 0.5pp MDE, and duration is derived from the 40k weekly traffic and 20k new-customer accumulation, running in full weeks.

Sonnet 5.5 · API

Sample size is calculated from the 8% baseline and 0.5pp effect, and duration is derived from the 40k weekly traffic and new-customer accumulation, running whole weeks.

Says why a test is worth it hereRightRightRight
Haiku 5.5 · API

The spec demonstrates the test is powered for the smallest change that matters, and the decision rules show the result would change the team's action, which justifies running the test.

Opus 5.5 · Claude

The spec explains the trade-off (trust vs. drop-off), the 12% support-contact problem, and confirms sufficient traffic to detect the smallest worthwhile effect.

Sonnet 5.5 · API

The spec demonstrates the test is feasible by sizing it to detect the smallest effect that matters within the available traffic, implicitly justifying the test.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 85% of 39 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Sonnet 5.5API92.696.42None
2Haiku 5.5API83.496.72None
3Opus 5.5Claude88.779.02None
4GPT-6 AstraChatGPT88.976.02None
5GPT-6 LunaAPI89.165.22None
6GPT-6.1 SolAPI83.169.32None
7Gemini 3.8 FlashAPI77.347.92None
8Gemini 3.5 Flash-LiteGemini47.813.321 capped

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.20304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreSonnet 5.5 · API: task score 95, 33× the cheapestHaiku 5.5 · API: task score 89, 2.7× the cheapestOpus 5.5 · Claude: task score 81, 49× the cheapestGPT-6 Astra · ChatGPT: task score 79, 79× the cheapestGPT-6 Luna · API: task score 76, the cheapestGPT-6.1 Sol · API: task score 75, 13× the cheapestGemini 3.8 Flash · API: task score 60, 7.4× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 29, 2.1× the cheapest, 1 critical failureSonnet 5.5 · APIHaiku 5.5 · APIOpus 5.5 · ClaudeGPT-6 Astra · ChatGPTGPT-6 Luna · APIGPT-6.1 Sol · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Nobody beats these on both score and costAt least one critical failure

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Specifying an A/B test before running it.

Why it matters

Most tests are underpowered, or measure a metric nobody agrees is good.

What good looks like

  • One primary metric everyone agrees on the direction of
  • Decision rule stated up front
  • Guardrails
  • Power considered

Deliberately not measured

    Capability tested

    Test design

    The failure we’re looking for

    A test with no decision attached

    Grading

    Decision model and LLM judge, calibrated against a blind PM review