Tasks / Metrics & Experimentation

Experiment specification

Can the model design a test that could actually change the decision?

Task type v1.0 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 31% were usable with at most a quick edit.

Reliably right

Randomises by user100% pass
Randomises by logged-in user ID across platforms, explicitly citing cross-platform usage and session non-independence.
Sonnet 5.5 · API · Show the fee sooner
Works around the calendar94% pass
Addresses both the payments test and the Black Friday freeze, proposes concurrent independent assignment with an interaction check, and avoids a one-week squeeze.
Sonnet 5.5 · API · Show the fee sooner
Addresses the actual decision91% pass
Commits to a clear experiment plan with a decision rule that maps every outcome (ship, don't ship, kill, extend) to an action.
Sonnet 5.5 · API · Show the fee sooner

Where it slips

Guardrails with thresholds50% pass
Guardrail metrics are named but lack explicit numeric thresholds; 'materially worse' and 'drops meaningfully' are too vague to block rollout without further definition.
Sonnet 5.5 · API · Showing the delivery fee up front
Says why a test is worth it here59% pass
The output does not explain why this decision merits a test given the traffic and stakes, nor does it recommend a cheaper alternative.
Opus 5.5 · Claude · Show the fee sooner
Uses the supplied evidence correctly64% pass
Unsupported claims about the traffic data's adequacy and the code freeze's preapproval requirement are presented as facts, not in the brief.
GPT-6.1 Sol · API · Show the fee sooner

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

An unambiguous primary metric
Does the spec name one primary metric, explain why an increase is unambiguously good for the user and the business, and say how the result's trustworthiness will be checked?
Passes when One primary metric with a clear rationale for its direction, plus a planned trust check such as a sample ratio check.
Ronny KohaviLenny’s Podcast
Decision rule written before the test
Does the spec say, before the test runs, which result leads to which action (ship, don't ship, iterate), including what happens if the result is inconclusive?
Passes when Every outcome, including an inconclusive one, maps to a stated action, with the thresholds that separate them.
Ronny KohaviLenny’s Podcast
Sized from the real traffic
Is the sample size and duration worked out from the supplied traffic, baseline and smallest effect worth detecting, run in full weekly cycles?
Passes when Sample size follows from the baseline and the smallest effect worth acting on, the duration follows from the traffic the brief gives (including how units accumulate), and it runs whole weeks.
Product and growth leadersOn Lenny’s Podcast
Guardrails with thresholds
Does the spec name the guardrail metrics that would stop a rollout even if the primary metric wins, each with a threshold?
Passes when The guardrails that matter for this change are named, each with a threshold that would block the rollout.
Lauryn IsfordLenny’s Podcast
Says why a test is worth it here
Does the spec weigh whether a test is worth running here: enough traffic to detect an effect that matters, and a decision the result would actually change, and does it say what it would do instead if not?
Passes when Explains why this decision merits a test given the traffic and stakes, or recommends a cheaper route where they don't.
Lauryn IsfordLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for checkout at Basketful. We want to test showing the delivery fee in the basket, instead of only at the last step of checkout. Write the experiment spec for Ravi, the growth engineer who'll build it, Ines, the analyst who'll read it out, and Chloe, our Head of Growth, who approves it. Keep it under 700 words. The team's draft plan is below. Fix what needs fixing.

What the model was given5 items: About Basketful, Why we're testing it, Traffic and baselines, Smallest change worth acting on, The team's draft plan

About BasketfulAn online grocery service. Delivery costs £3.99, or is free on orders over £60. Today the fee first appears on the final checkout step.

Why we're testing it12% of support contacts last quarter were about delivery fees the customer hadn't expected. Design believes showing the fee early will build trust; Chloe worries it will put people off before they start checkout.

Traffic and baselinesAbout 40,000 different customers view their basket each week. About half of each week's basket viewers are new that week; the rest came back from earlier weeks. 8% of basket viewers place an order. Of customers who start checkout, 62% complete it. Average order value: £47.

Smallest change worth acting onChloe: “Anything smaller than half a percentage point on orders isn't worth arguing about.”

The team's draft planPrimary metric: checkout completion rate (orders ÷ checkout starts). Split: 50/50 by session. Duration: we'll check the dashboard every day and stop as soon as it's significant. Chloe's addition: “While we're at it, let's put the free-delivery-over-£60 banner in the variant too.”

What a strong answer doesThe answer key the graders mark against

A spec that tests one change and fixes the draft's three problems. The primary metric is orders per basket-viewing customer, not checkout completion rate: showing the fee earlier changes who starts checkout, so completion can rise while orders fall. Customers, not sessions, are randomised, so returning customers see one version. The free-delivery banner stays out of the variant (or gets its own test), or the result can't be attributed. The duration is fixed in advance, with no daily stopping: detecting 0.5pp on an 8% base needs roughly 47,000 customers per arm (about 95,000 in total), and because half of each week's viewers are returning customers, new customers accumulate at about 40,000, 60,000, 80,000 and 100,000 after four weeks, so it runs about four full weeks, not two and a half. Guardrails with thresholds: average order value, fee-related support contacts, and the checkout-start rate as a diagnostic. A sample ratio check before reading results. A decision rule written up front: ship if orders are no worse than 0.5pp down and fee contacts fall; don't ship if orders drop by 0.5pp or more; with a stated plan for an inconclusive result.

Critical failures (cap the score at 40)

Uses checkout completion rate (orders ÷ checkout starts) as the primary metric
Bundles the free-delivery banner into the variant

Task

Mid-level · v1.3

The outputs

8 setups answered this brief.

Sonnet 5.5API93Usable with a quick edit Haiku 5.5API93Usable with a quick edit GPT-6 AstraChatGPT80Needs more work GPT-6.1 SolAPI79Usable with a quick edit GPT-6 LunaAPI77Usable with a quick edit Opus 5.5Claude75Needs more work Gemini 3.8 FlashAPI48Needs more work Gemini 3.5 Flash-LiteGemini18Critical failure

Haiku 5.5API

Usable with a quick edit12 of 13 checks passed

The spec is complete, directly addresses all team members, and a PM could use it as-is, only needing to fill in the two baselines that are clearly flagged.

Output

Experiment Spec: Delivery Fee in Basket

To: Ravi (build), Ines (analysis), Chloe (approval) From: PM, Checkout Status: For approval

What changed from the draft

Primary metric moved from checkout completion to basket-to-order conversion. Completion only measures people who reach checkout, so it would miss the risk Chloe raised: people leaving before checkout.
Randomise by customer, not session. Half of each week's viewers are returning customers. Session splits would show the same person both versions.
Fixed duration, pre-set sample size. Checking daily and stopping at first significance inflates false positives.
Free-delivery banner removed. It changes the same thing we're measuring, so we couldn't attribute any effect. We'll propose it as a separate test.

Hypothesis

Showing the delivery fee in the basket changes basket-to-order conversion. We expect an increase of at least 0.5 percentage points (8.0% to 8.5%). The test is two-sided, so a decrease would also be recorded and acted on.

Variants

Control: Basket as today. Fee appears only on the final checkout step.
Variant: Basket shows a delivery line under the subtotal: "Delivery: £3.99" or "Delivery: free" once the basket is over £60. Checkout is unchanged.

Ravi: The line must update live as the basket crosses the £60 threshold, and must match the fee charged at checkout exactly.

Population and randomisation

All customers who view their basket during the test period.
Randomise 50/50 by customer ID at first basket view in the test. Assignment stays fixed for the whole test.
Run a sample ratio mismatch (SRM) check on day 1. If the split deviates from 50/50 at p < 0.01, pause and investigate before continuing.

Metrics

Primary: Basket-to-order conversion. Customers who place an order within 7 days of their first basket view in the test, divided by customers who viewed their basket. Baseline: 8%.

Guardrails: - Checkout start rate (checkout starts ÷ basket viewers). Ines to pull baseline. - Checkout completion rate. Baseline: 62%. - Average order value. Baseline: £47. - Delivery-fee support contacts per 1,000 orders. Ines to pull baseline; currently 12% of all contacts.

Sample size and duration

Baseline 8.0%, minimum effect 0.5 percentage points absolute, two-sided α = 0.05, power 80%.
Requires about 47,500 customers per arm (95,000 total).
Traffic is about 40,000 basket viewers per week, of whom about 20,000 are new. Unique customers therefore accumulate at roughly 40k in week 1 and 20k per week after that.
Plan: 4 weeks of enrolment, then a 7-day conversion window, for a fixed end date of [date]. Ines to confirm the unique-customer count, since the 20k-new-per-week figure is an estimate.
No decisions before the end date, with one exception: at week 2, Ines runs a harm-only check on guardrails. We stop early only if a guardrail shows harm at p < 0.001.

Decision rules

Ship if: - The primary metric is significantly positive (p < 0.05), with an estimated uplift of at least 0.5 percentage points, and - No guardrail is significantly worse.

Don't ship if: - The primary result is not significant, or - It is significant but below 0.5 points (not worth acting on, per Chloe), or - A guardrail is significantly worse.

We don't iterate on a null result by re-slicing the data until something looks significant.

Analysis (Ines)

Analyse at the customer level, intention-to-treat: all assigned customers, regardless of whether they saw the fee line.
Report absolute difference in percentage points and relative difference, each with a 95% confidence interval.
Report all guardrails.
Pre-registered segment: new vs returning customers. Returning customers may already know the fee, so the effect may differ. Treat any other segment cuts as exploratory.

Timeline and sign-off

Build complete: [date]
Launch: [date]
Fixed end: [date]
Readout: [date + 3 working days]

Approved by: _____ Chloe, Head of Growth Date: ___

Grades and run details

Decision model 85 · LLM judge 100

Decision model checks

failUses the supplied evidence correctly33%
passAddresses the actual decision79%
passRespects explicit constraints14%
passIdentifies material uncertainty45%
passAvoids unsupported claims28%
passProduces the required deliverable34%
passTests one change at a time95%
passFixes the duration up front97%
passAn unambiguous primary metric77%
passDecision rule written before the test41%
partialSized from the real traffic40%
partialGuardrails with thresholds91%
passSays why a test is worth it here62%

Run

Run: #1
API response time: 29 s
Submitted: 8 Oct 2026

GPT-6 AstraChatGPT

Needs more than a quick edit11 of 13 checks passed

Guardrail thresholds are missing and the spec defers their definition, so a PM would need to add them before the spec is actionable.

Output

Experiment: show delivery fees in the basket

Owner/build: Ravi · Analysis: Ines · Approval: Chloe Decision: Should Basketful show delivery charges before checkout?

Hypothesis and scope

Showing the fee in the basket will reduce fee surprises, but could discourage ordering. Unexpected delivery fees accounted for 12% of support contacts last quarter; this is motivation, not a baseline customer complaint rate.

Test fee visibility only. Do not add the proposed free-delivery promotional banner: it introduces a second change and would prevent us attributing results to earlier disclosure.

Experience

Control: Current experience; delivery fee first appears at the final checkout step.
Treatment: Show a clearly labelled delivery line beside the basket total: £3.99 delivery, or Free delivery when eligible. Update the fee and total as the basket changes.
Use the existing pricing service and eligibility rules. Confirm behaviour at exactly £60 before launch; “over £60” must not silently become “£60 or more.”
Leave checkout, pricing and other messaging unchanged.

Eligibility and assignment

Include customers viewing a non-empty, orderable basket. Exclude staff, bots and test accounts using rules fixed before launch.

Randomise 50/50 by customer, not session, at their first eligible basket view. Persist assignment across visits and devices where identity is available. For signed-out customers, use a stable first-party identifier; Ravi must document identity merging and cross-device limitations before launch.

Analyse everyone assigned, whether or not they start checkout or successfully see the fee: intention to treat.

Metrics

Primary: Customer order conversion — percentage of assigned customers placing at least one order within seven days of assignment. Report treatment minus control in percentage points, with a two-sided 95% confidence interval.

Checkout completion is diagnostic only: treatment may change who starts checkout, so orders ÷ checkout starts could improve while overall ordering falls.

Secondary/diagnostic: - Checkout-start rate and checkout completion. - Delivery-fee-related support contacts per assigned customer within seven days, using consistent tagging. - Orders and revenue per assigned customer; average order value and free-delivery eligibility as diagnostics.

Guardrails: Contribution margin per assigned customer, if reliably available; otherwise net revenue per assigned customer as an explicitly limited proxy. Also monitor basket/checkout errors. Chloe and Ines must approve numerical economic and reliability tolerances before launch.

Sample size and schedule

Chloe’s smallest worthwhile order-conversion change is 0.5 percentage points: 8.0% to 8.5%, not a 0.5% relative lift.

At 5% two-sided significance and 80% power, detecting that increase requires approximately 48,000 customers per arm, or 96,000 total.

Ines must first verify that the 8% baseline matches the eligibility rules and seven-day outcome window; recalculate and freeze the sample target if necessary.

Do not count returning customers again. Expected recruitment is approximately:

Week 1: 40,000 unique customers.
Each subsequent week: 20,000 additional customers.
Four weeks: approximately 100,000 unique customers.

Enroll for at least four complete weeks and until the frozen sample target is reached, ending at a full-week boundary. Then wait seven days for the final cohort’s outcomes: roughly five weeks to readout.

Monitoring and decision

Ravi validates assignment persistence, fee accuracy, exposure/order linkage and sample-ratio balance before ramp-up.

Daily checks are for instrumentation and predefined safety breaches—not efficacy stopping. No “stop when significant,” sample extensions based on results, or repeated significance decisions.

Ines delivers one final primary analysis. Chloe’s default rollout rule is: - Estimated conversion gain at least +0.5 points; - 95% confidence interval excludes zero; - Guardrails pass.

Otherwise, do not roll out on this result alone. A non-significant result does not prove no harm. Secondary trust improvements inform follow-up, not a post-hoc replacement for the primary success rule.

Grades and run details

Decision model 88 · LLM judge 79

Decision model checks

passUses the supplied evidence correctly5%
passAddresses the actual decision46%
partialRespects explicit constraints29%
passIdentifies material uncertainty73%
passAvoids unsupported claims69%
passProduces the required deliverable39%
passTests one change at a time98%
passFixes the duration up front84%
passAn unambiguous primary metric73%
partialDecision rule written before the test26%
passSized from the real traffic54%
partialGuardrails with thresholds98%
passSays why a test is worth it here84%

Run

Run: #1
Time to output: 47 s
Submitted: 1 Oct 2026

GPT-6.1 SolAPI

Usable with a quick edit9 of 13 checks passed

The spec is well-structured and fixes the draft's main problems; a PM could add guardrail thresholds and a brief justification with a quick edit.

Output

Experiment spec: delivery-fee visibility in basket

Owner: Checkout PM Build: Ravi · Analysis: Ines · Approval: Chloe

Objective and hypothesis

Test whether showing delivery costs before checkout increases orders by reducing surprises—or decreases orders by discouraging customers earlier.

Last quarter, 12% of support contacts concerned unexpected delivery fees. This motivates the test but is not the percentage of customers affected.

Experience

Control: Current basket; delivery fee first appears at the final checkout step.

Variant: Show a clearly labelled delivery-fee line and fee-inclusive total in the basket. Update both when the basket changes. Use the production pricing logic: £3.99 delivery, free for orders over £60. Ravi must verify boundary behaviour at exactly £60 and match existing checkout pricing.

Keep the final checkout fee display unchanged. Everything else stays identical.

Do not add the free-delivery banner. It could independently encourage larger baskets, making it impossible to attribute results to earlier fee visibility. Test it separately.

Eligibility and assignment

Include customers viewing an orderable basket in supported markets. Apply existing employee, test-account and bot exclusions identically, before assignment.

Randomise 50/50 by customer, not session, at their first eligible basket view. Persist assignment across visits and devices where identity is available. Use a stable anonymous identifier for guests and a documented account-linking rule; Ravi and Ines must validate identity handling before launch.

Each customer enters the experiment once. Analyse everyone assigned, regardless of whether they start checkout or successfully see the treatment.

Metrics

Primary: Seven-day order conversion: customers placing at least one order within seven days of assignment ÷ all assigned eligible basket viewers.

This captures Chloe’s concern about discouraging checkout entry. Orders ÷ checkout starts is unsuitable as primary: the treatment can change who starts checkout.

Secondary/diagnostic: - Checkout-start rate per assigned customer. - Checkout completion among starters, interpreted cautiously because starter populations may differ. - Revenue per assigned customer, including non-buyers. - AOV and proportion of orders qualifying for free delivery. - Unexpected-fee support contacts per assigned customer within seven days, using consistent contact classification.

Operational guardrails: Basket/checkout errors, incorrect fee calculations and page performance. These can trigger a safety rollback, not an early declaration of success.

Sample size and duration

Interpret Chloe’s minimum worthwhile change as 0.5 percentage points absolute: 8.0% to 8.5%, not a 0.5% relative lift.

With a matching 8% baseline, two-sided 5% significance and 80% power, plan for approximately 95,000 unique customers total, evenly split. Ines must verify the historical baseline uses the same seven-day definition and finalise the calculation before launch.

Do not count 40,000 weekly viewers as entirely new experiment entrants. Estimated cumulative reach is 40,000 in week one, then roughly 20,000 additional customers weekly: about four weeks to enrol the target, plus seven days for outcomes to mature.

Lock the enrolment target before launch; stop enrolment when reached, then wait seven days. Extend only if unique-customer accrual is slower—not based on results.

Analysis and decision

Ines reports the intention-to-treat conversion difference, 95% confidence interval and absolute rates. Check allocation imbalance, logging, identity consistency and treatment exposure before interpreting results.

No daily significance-based stopping. Daily dashboards are for operational safety only.

Recommend rollout if the primary improvement is statistically significant, the point estimate is at least +0.5 percentage points, and there are no unresolved operational or economic concerns.

Otherwise, retain control pending review. Distinguish evidence of harm, evidence ruling out a worthwhile gain, and an inconclusive result; “not significant” does not mean “no effect.”

Launch approval

Ravi signs off implementation and QA; Ines signs off measurement and the locked analysis plan; Chloe approves the isolated treatment, decision rule and any economic guardrail thresholds before launch.

Grades and run details

Decision model 77 · LLM judge 79

Decision model checks

failUses the supplied evidence correctly6%
passAddresses the actual decision66%
partialRespects explicit constraints19%
passIdentifies material uncertainty38%
passAvoids unsupported claims68%
passProduces the required deliverable57%
passTests one change at a time100%
passFixes the duration up front74%
passAn unambiguous primary metric80%
partialDecision rule written before the test42%
partialSized from the real traffic41%
partialGuardrails with thresholds98%
passSays why a test is worth it here88%

Run

Run: #1
API response time: 36 s
Submitted: 30 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 4

Uses the supplied evidence correctlyMixedRightMixed

Haiku 5.5 · API

All statements about the current situation are directly from the supplied context or simple arithmetic; no facts are invented.

GPT-6 Astra · ChatGPT

All factual claims about the current situation are taken from the brief or follow by arithmetic; the pricing service is a reasonable inference from the delivery fee rules.

GPT-6.1 Sol · API

All factual claims about the current situation are taken directly from the brief or derived correctly from it.

Fixes the duration up frontRightRightMixed

Haiku 5.5 · API

The spec sets a fixed 4-week enrolment plus 7-day conversion window, explains that daily peeking inflates false positives, and only allows a harm-based early stop on guardrails.

GPT-6 Astra · ChatGPT

Sets a fixed duration of at least four full weeks, states daily checks are for safety only, and prohibits stopping on significance.

GPT-6.1 Sol · API

It sets a fixed duration but does not explain why daily peeking inflates false positives; it only states not to do it.

Guardrails with thresholdsRightWrongWrong

Haiku 5.5 · API

Checkout start rate, completion rate, average order value, and delivery-fee support contacts are listed as guardrails, with a requirement that they are not significantly worse for rollout to proceed.

GPT-6 Astra · ChatGPT

Guardrail metrics are named but no numerical thresholds are set; the spec defers approval of tolerances, leaving the spec incomplete.

GPT-6.1 Sol · API

It lists guardrail metrics but does not set specific thresholds (e.g., for AOV or support contacts) that would block rollout.

Says why a test is worth it hereRightMixedMixed

Haiku 5.5 · API

The spec demonstrates the test is powered for the smallest change that matters, and the decision rules show the result would change the team's action, which justifies running the test.

GPT-6 Astra · ChatGPT

Does not explain why a test is worth running given the traffic and stakes; it assumes the test is justified without weighing alternatives.

GPT-6.1 Sol · API

It does not explain why this test is worth running given the traffic and stakes, nor does it recommend a cheaper alternative.

All got right 9

Addresses the actual decisionRightRightRight

Haiku 5.5 · API

The output commits to a clear experimental design and includes decision rules that map results to specific actions (ship/don't ship), which addresses the team's need to decide how to test and what to act on.

GPT-6 Astra · ChatGPT

Commits to a clear answer (run the experiment, roll out only if conversion gain ≥0.5pp, CI excludes zero, guardrails pass) and states what would change it.

GPT-6.1 Sol · API

The spec commits to a clear experiment plan with a decision rule for rollout, addressing the request for an experiment spec.

Respects explicit constraintsRightRightRight

Haiku 5.5 · API

The spec is within the 700-word limit, is addressed to Ravi, Ines, and Chloe, and proposes an actionable test plan.

GPT-6 Astra · ChatGPT

The spec is under 700 words, addresses the named readers, and is in the requested experiment-spec form.

GPT-6.1 Sol · API

The output is under 700 words, addresses the named readers, and fixes the draft's problems as required.

Identifies material uncertaintyRightRightRight

Haiku 5.5 · API

The output names uncertainty in the new-customer accumulation estimate and asks Ines to confirm, and notes the segment effect may differ; it bounds where possible.

GPT-6 Astra · ChatGPT

Names specific unknowns (baseline verification, guardrail tolerances, identity merging limitations) and says how they will be resolved before launch.

GPT-6.1 Sol · API

It identifies unknowns (baseline verification, identity handling, enrollment speed) and says how to resolve them, and distinguishes inconclusive results.

Avoids unsupported claimsRightRightRight

Haiku 5.5 · API

Hypotheses, forecasts, and reasoning are clearly labelled as such; no confident claims go beyond the supplied evidence.

GPT-6 Astra · ChatGPT

Hypotheses are labelled as such, and confident claims are supported by the brief.

GPT-6.1 Sol · API

Hypotheses and interpretations are clearly labeled, and no confident claims go beyond the supplied evidence.

Produces the required deliverableRightRightRight

Haiku 5.5 · API

The output is a complete experiment spec with sections for build, analysis, and sign-off, and is usable by the intended readers with minimal editing.

GPT-6 Astra · ChatGPT

The experiment spec is complete, under 700 words, and usable by Ravi, Ines, and Chloe with light edits.

GPT-6.1 Sol · API

The output is a complete experiment spec in the requested form, under 700 words, and usable by the named readers.

Tests one change at a timeRightRightRight

Haiku 5.5 · API

The free-delivery banner is explicitly removed from the variant with a clear explanation that it would confound attribution, and it is suggested as a separate test.

GPT-6 Astra · ChatGPT

Explicitly excludes the free-delivery banner from the variant and explains that adding it would prevent attribution.

GPT-6.1 Sol · API

It explicitly excludes the free-delivery banner and explains that bundling would make attribution impossible.

An unambiguous primary metricRightRightRight

Haiku 5.5 · API

A single primary metric (basket-to-order conversion) is named with a clear rationale for its direction, and a sample ratio mismatch check is planned.

GPT-6 Astra · ChatGPT

Names one primary metric (customer order conversion), explains why an increase is unambiguously good, and includes a sample-ratio balance check.

GPT-6.1 Sol · API

It names one primary metric (seven-day order conversion) with a clear rationale and includes a trust check (allocation imbalance, logging, etc.).

Decision rule written before the testRightRightRight

Haiku 5.5 · API

Every possible result is mapped to an action: ship if significant with ≥0.5pp uplift and guardrails safe; don't ship otherwise; no iteration on null results.

GPT-6 Astra · ChatGPT

States a pre-defined rule: ship if conversion gain ≥0.5pp, CI excludes zero, guardrails pass; otherwise do not roll out, including for inconclusive results.

GPT-6.1 Sol · API

It states rollout conditions (significant, ≥0.5pp, no concerns) and what to do otherwise, including handling inconclusive results.

Sized from the real trafficRightRightRight

Haiku 5.5 · API

Sample size is correctly derived from the 8% baseline, 0.5pp effect, 80% power; duration follows from the 40k weekly traffic and half-new accumulation, running whole weeks.

GPT-6 Astra · ChatGPT

Sample size (48k/arm) is calculated from the 8% baseline and 0.5pp MDE; duration uses the traffic data (40k unique/week, half new) and runs whole weeks.

GPT-6.1 Sol · API

Sample size is calculated from the 8% baseline and 0.5pp effect, and duration follows from the traffic accumulation pattern, running whole weeks.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 85% of 39 checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	Sonnet 5.5withAPI	92.6	96.4	2	None
2	Haiku 5.5withAPI	83.4	96.7	2	None
3	Opus 5.5withClaude	88.7	79.0	2	None
4	GPT-6 AstrawithChatGPT	88.9	76.0	2	None
5	GPT-6 LunawithAPI	89.1	65.2	2	None
6	GPT-6.1 SolwithAPI	83.1	69.3	2	None
7	Gemini 3.8 FlashwithAPI	77.3	47.9	2	None
8	Gemini 3.5 Flash-LitewithGemini	47.8	13.3	2	1 capped

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Nobody beats these on both score and costAt least one critical failure

Task score and relative cost by setup
Setup	Task score	Relative cost (cheapest = 1)
GPT-6 Luna · API	76	1.0×
Gemini 3.5 Flash-Lite · Gemini	29	2.1×
Haiku 5.5 · API	89	2.7×
Gemini 3.8 Flash · API	60	7.4×
GPT-6.1 Sol · API	75	13×
Sonnet 5.5 · API	95	33×
Opus 5.5 · Claude	81	49×
GPT-6 Astra · ChatGPT	79	79×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Specifying an A/B test before running it.

Why it matters

Most tests are underpowered, or measure a metric nobody agrees is good.

What good looks like

One primary metric everyone agrees on the direction of
Decision rule stated up front
Guardrails
Power considered

Deliberately not measured

Capability tested

Test design

The failure we’re looking for

A test with no decision attached

Grading

Decision model and LLM judge, calibrated against a blind PM review

What AI gets right here, and what you’ll still have to catch

Reliably right

Where it slips

How it’s graded

An unambiguous primary metric

Decision rule written before the test

Sized from the real traffic

Guardrails with thresholds

Says why a test is worth it here

Plus our standard checks

The tasks

The outputs

Output

Experiment Spec: Delivery Fee in Basket

What changed from the draft

Hypothesis

Variants

Population and randomisation

Metrics

Sample size and duration

Decision rules

Analysis (Ines)

Timeline and sign-off

Grades and run details

Decision model checks

Run

Output

Experiment: show delivery fees in the basket

Hypothesis and scope

Experience

Eligibility and assignment

Metrics

Sample size and schedule

Monitoring and decision

Grades and run details

Decision model checks

Run

Output

Experiment spec: delivery-fee visibility in basket

Objective and hypothesis

Experience

Eligibility and assignment

Metrics

Sample size and duration

Analysis and decision

Launch approval

Grades and run details

Decision model checks

Run

Check by check

Where they differ 4

All got right 9

Results

What you get for the cost

About the task

The PM job

Why it matters

What good looks like

Deliberately not measured