Humanity’s Last PRD
Latest drop: Gemini 3.8 Flash · 1 Oct
Tasks / Experiment

Experiment specification

Can the model design a test that could actually change the decision?

Measures the modelTask v1.0 · 2 casesDifficulty

What AI gets right here, and what you’ll still have to catch

From 14 graded outputs by 7 models. 36% were usable with at most a quick edit.

Reliably right

  1. Addresses the actual decision91% pass
    The output commits to a clear decision rule: launch only if all gates pass, otherwise do not launch, and an inconclusive result is not a pass.
    GPT-6.1 Sol · API · Batching deliveries before peak season
  2. Tests one change at a time86% pass
    It explicitly keeps the free-delivery banner out of the variant and explains that adding it would make the result uninterpretable.
    GPT-6 Luna · API · Showing the delivery fee up front
  3. Identifies material uncertainty84% pass
    It identifies unknowns like power under correlation, carryover, and city differences, and states that if power is insufficient the rollout will not proceed, resolving the uncertainty.
    GPT-6.1 Sol · API · Batching deliveries before peak season

Where it slips

  1. Respects explicit constraints55% pass
    The output is approximately 780 words, exceeding the 'under 700 words' limit.
    Opus 5.5 · Claude · Showing the delivery fee up front
  2. Sized from the real traffic57% pass
    Sample size is correct, but the test stops when the enrolment target is reached rather than running fixed whole weeks.
    GPT-6.1 Sol · API · Showing the delivery fee up front
  3. Guardrails with thresholds59% pass
    Guardrail metrics are listed but no specific thresholds (e.g., maximum acceptable drop in AOV) are given to block rollout.
    GPT-6 Luna · API · Showing the delivery fee up front

Case viewer

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for checkout at Basketful. We want to test showing the delivery fee in the basket, instead of only at the last step of checkout. Write the experiment spec for Ravi, the growth engineer who'll build it, Ines, the analyst who'll read it out, and Chloe, our Head of Growth, who approves it. Keep it under 700 words. The team's draft plan is below. Fix what needs fixing.

What the model was given5 items: About Basketful, Why we're testing it, Traffic and baselines, Smallest change worth acting on, The team's draft plan
About BasketfulAn online grocery service. Delivery costs £3.99, or is free on orders over £60. Today the fee first appears on the final checkout step.
Why we're testing it12% of support contacts last quarter were about delivery fees the customer hadn't expected. Design believes showing the fee early will build trust; Chloe worries it will put people off before they start checkout.
Traffic and baselinesAbout 40,000 different customers view their basket each week. About half of each week's basket viewers are new that week; the rest came back from earlier weeks. 8% of basket viewers place an order. Of customers who start checkout, 62% complete it. Average order value: £47.
Smallest change worth acting onChloe: “Anything smaller than half a percentage point on orders isn't worth arguing about.”
The team's draft planPrimary metric: checkout completion rate (orders ÷ checkout starts). Split: 50/50 by session. Duration: we'll check the dashboard every day and stop as soon as it's significant. Chloe's addition: “While we're at it, let's put the free-delivery-over-£60 banner in the variant too.”
What a strong answer doesThe answer key the graders mark against

A spec that tests one change and fixes the draft's three problems. The primary metric is orders per basket-viewing customer, not checkout completion rate: showing the fee earlier changes who starts checkout, so completion can rise while orders fall. Customers, not sessions, are randomised, so returning customers see one version. The free-delivery banner stays out of the variant (or gets its own test), or the result can't be attributed. The duration is fixed in advance, with no daily stopping: detecting 0.5pp on an 8% base needs roughly 47,000 customers per arm (about 95,000 in total), and because half of each week's viewers are returning customers, new customers accumulate at about 40,000, 60,000, 80,000 and 100,000 after four weeks, so it runs about four full weeks, not two and a half. Guardrails with thresholds: average order value, fee-related support contacts, and the checkout-start rate as a diagnostic. A sample ratio check before reading results. A decision rule written up front: ship if orders are no worse than 0.5pp down and fee contacts fall; don't ship if orders drop by 0.5pp or more; with a stated plan for an inconclusive result.

Critical failures (cap the score at 40)
  • Uses checkout completion rate (orders ÷ checkout starts) as the primary metric
  • Bundles the free-delivery banner into the variant
Case

Mid-level · v1.1 · synthetic · B2C, ecommerce, pricing, metric choice

The outputs

7 setups answered this brief. Pick up to three to read side by side.

GPT-6 LunaAPI83Usable with a quick editGPT-6.1 SolAPI81Usable with a quick editSonnet 5.5API79Needs more workOpus 5.5Claude73Needs more workGPT-6 AstraChatGPT71Needs more workGemini 3.8 FlashAPI42Needs more workGemini 3.5 Flash-LiteGemini23Critical failure
Gemini 3.5 Flash-LiteGemini
Critical failure: Bundles the free-delivery banner into the variant. Score capped at 40.

Needs more than a quick edit1 of 12 checks passed

The spec bundles the banner, uses an inadequate duration, and lacks a decision rule and guardrail thresholds, so a PM would need to redo core parts.

Output

To: Chloe (Head of Growth), Ravi (Growth Engineer), Ines (Data Analyst) From: PM, Checkout Subject: Experiment Spec: Early Delivery Fee Display in Basket

Team, here is the polished spec for our upcoming checkout experiment. I’ve incorporated Chloe’s threshold, fixed a few methodological traps in the draft, and scoped this for a clean, reliable read.

---

1. Objective & Hypothesis

  • Current State: Delivery fee (£3.99, free over £60) is only revealed on the final checkout step. 12% of last quarter’s support contacts were fee-related surprises.
  • Hypothesis: Showing the delivery fee (and progress to free delivery) directly in the basket increases customer trust and sets clear expectations upfront.
  • Expected Impact: While Chloe rightly worries this might cause some upfront drop-off among sub-£60 baskets, we believe the resulting higher-intent checkout starts will lift overall conversion and reduce cart abandonment due to "sticker shock."

---

2. Variant Definition

  • Control (A): Existing basket experience. Delivery fee hidden until final checkout step.
  • Variant (B):
  • Displays the delivery fee clearly in the basket summary.
  • Note on scope: Per Chloe’s suggestion, we are bundling the "free delivery over £60" progress banner into Variant B. Since the fee and the threshold are conceptually tied, showing one without the context of the other would skew the test.

---

3. Traffic Allocation & Unit of Randomization

  • Unit of Randomization: User-level (logged-in state or persistent cookie). Correction from draft: Session-level splitting would cause a jarring experience if a user refreshes or returns and sees fluctuating fee visibility.
  • Split: 50/50 (Control / Variant).
  • Audience: All customers viewing the basket (~40,000 unique users/week, split roughly 50/50 between new and returning).

---

4. Metrics

  • Primary Metric (Guardrail for Business Value):
  • Order Conversion Rate: Completed Orders ÷ Total Basket Views.
  • (Correction from draft: The draft proposed "Checkout Completion Rate (Orders ÷ Checkout Starts)." That misses top-of-funnel drop-off. If Variant B scares people away before checkout, measuring only checkout-starts will falsely make the variant look better. We must measure from the basket view).
  • Secondary Metrics:
  • Checkout Start Rate: Checkout Starts ÷ Basket Views.
  • Checkout Completion Rate: Orders ÷ Checkout Starts (to isolate downstream behavior).
  • Average Order Value (AOV): To monitor if the £60 threshold banner successfully drives basket-building behavior (baseline: £47).
  • Support Metric:
  • Volume of delivery-fee-related support tickets (Ines, let’s pull this baseline post-launch).

---

5. Success Criteria & Statistical Guardrails

  • Minimum Detectable Effect (MDE): Per Chloe, we care about a ≥ +0.5 percentage point absolute change in the Primary Metric (Order Conversion Rate from basket view, baseline ~8%).
  • Duration & Peeking:
  • Correction from draft: Do not stop the test the moment p < 0.05. Early peeking yields false positives.
  • We will run the test for a fixed duration of 2 full weeks to capture day-of-week and new vs. returning user cycles (~80,000 total unique users), yielding adequate statistical power for our MDE.

---

Next Steps:

  • Ravi: Let’s sync on tracking implementation for the basket-level fee and banner. Target build completion: [Insert Date].
  • Ines: Please set up the tracking dashboard, ensuring we segment by New vs. Returning users and Basket Value (<£60 vs. >£60).
  • Chloe: Good to approve?

Claims the judge couldn’t find in the brief

  • 2 full weeks yields ~80,000 total unique users and adequate statistical power for the MDE.

Grades and run details

Decision model 33 · LLM judge 2 of 13 checks
Decision model checks
  • failUses the supplied evidence correctly10%
  • failAddresses the actual decision65%
  • failRespects explicit constraints16%
  • partialIdentifies material uncertainty73%
  • partialAvoids unsupported claims30%
  • partialProduces the required deliverable75%
  • failTests one change at a time99%
  • partialFixes the duration up front76%
  • partialAn unambiguous primary metric69%
  • partialDecision rule written before the test47%
  • partialSized from the real traffic69%
  • partialGuardrails with thresholds45%
Run
Run
#1
Time to output
5 s
Submitted
30 Sept 2026
Gemini 3.8 FlashAPI

Needs more than a quick edit3 of 12 checks passed

A PM would need to redo the sizing and duration, remove or redesign the early kill rule, add a sample-ratio check, and complete guardrail thresholds. These are substantive reasoning changes, not quick wording edits.

Output

Experiment Spec: Delivery Fee Transparency in Basket

To: Chloe (Approval), Ravi (Build), Ines (Analysis) From: Checkout PM

---

1. Objective & Hypothesis

  • Context: 12% of support contacts complain about surprise delivery fees at final checkout.
  • Hypothesis: Displaying the delivery fee (£3.99, or "Free" if >£60) upfront in the basket will set accurate expectations, reduce support contacts, and build customer trust without reducing net orders.
  • Risk (Chloe’s concern): Upfront cost shock could cause drop-off before checkout begins.

---

2. Corrections to the Draft Plan

1. Primary Metric Changed: Draft used Checkout Completion Rate (Orders ÷ Starts). Correction: Use Basket-to-Order Conversion Rate (Orders ÷ Basket Viewers). Measuring only checkout starts introduces severe selection bias—if fee visibility deters low-intent users at the basket, checkout completion would artificially increase while overall orders plummet. 2. Unit of Randomization Changed: Draft used session-level. Correction: Randomize by User ID (falling back to persistent anonymous visitor ID). 50% of weekly viewers are repeat shoppers; session-level randomization would expose returning users to conflicting experiences. 3. No Peeking / Fixed Duration: Draft suggested stopping once "significant." Correction: Continuous monitoring with early stopping inflates false-positive rates to >30%. We will fix sample size upfront and run for complete weekly cycles. 4. Scope Control (Chloe’s Banner): Cut the promotional free-delivery banner from this test. Adding a promotional banner introduces a confounding variable, making it impossible to isolate whether fee transparency helped or hurt. We will test the banner in a follow-up experiment.

---

3. Variants (For Ravi)

  • Allocation: 50/50 split at User/Persistent Device ID level. Triggered on basket view.
  • Control (A): Current basket. No delivery fee shown (fee appears on final checkout step as today).
  • Variant (B): Add a delivery fee line item to the Basket Order Summary:
  • Basket < £60: `Delivery: £3.99`
  • Basket ≥ £60: `Delivery: FREE`
  • Include delivery fee in the displayed Basket Total.
  • Note: Do not add promotional progress bars or marketing banners.

---

4. Metrics Framework (For Ines)

  • Primary Metric: Basket-to-Order Conversion Rate (Total Unique Ordering Users ÷ Total Unique Basket Viewers).
  • Guardrail Metrics:
  • Average Order Value (AOV baseline: £47).
  • Basket-to-Checkout Start Rate (tracks where abandonment occurs).
  • Total Net Revenue (Orders × AOV).
  • Secondary / Diagnostic Metrics:
  • Delivery-fee-related customer support ticket rate per 1,000 orders.
  • Checkout Completion Rate (Checkout Starts → Orders).

---

5. Sample Size & Run Time (For Ines & Chloe)

  • Baseline Conversion: 8.0% of basket viewers place an order.
  • Minimum Detectable Effect (MDE): Absolute ±0.5 percentage points (7.5% or 8.5%, a 6.25% relative change), per Chloe's threshold.
  • Statistical Parameters: $\alpha = 0.05$ (two-tailed), Power ($1 - \beta$) = 80%.
  • Required Sample: ~47,000 unique users per variant (~94,000 total unique basket viewers).
  • Duration: 3 full weeks (21 days).
  • With 40,000 viewers/week (~20k new, ~20k repeat), 3 weeks yields ~80,000–90,000 unique basket viewers across three full grocery-shopping cycles.
  • Ines will run the read-out strictly after the 21-day window closes.

---

6. Decision & Rollout Criteria (For Chloe)

  • Ship Variant: Primary conversion does not drop by more than Chloe’s threshold of -0.5pp (non-inferiority) AND delivery-related support contacts decrease significantly.
  • Iterate / Re-test: Basket-to-order conversion drops by >0.5pp. In this scenario, early fee disclosure hurts conversion, justifying Chloe’s concern; we then test the £60 threshold banner/progress bar to offset drop-off.
  • Kill Immediately: Overall revenue drops >5% statistically significantly at the end of Week 1 (safety check only; not an early win stop).

Claims the judge couldn’t find in the brief

  • Three weeks yields about 80,000–90,000 unique basket viewers.
  • Early stopping inflates false-positive rates to more than 30%.
  • The variant's free-delivery threshold is £60 or more.

Grades and run details

Decision model 67 · LLM judge 3 of 13 checks
Decision model checks
  • failUses the supplied evidence correctly55%
  • passAddresses the actual decision63%
  • partialRespects explicit constraints20%
  • passIdentifies material uncertainty65%
  • partialAvoids unsupported claims49%
  • partialProduces the required deliverable32%
  • passTests one change at a time100%
  • passFixes the duration up front66%
  • passAn unambiguous primary metric28%
  • partialDecision rule written before the test79%
  • partialSized from the real traffic94%
  • partialGuardrails with thresholds90%
Run
Run
#1
API response time
19 s
Submitted
1 Oct 2026
Sonnet 5.5API

Needs more than a quick edit10 of 12 checks passed

Guardrail thresholds are vague ('materially worse', 'drops meaningfully'), so a PM would need to define them before the test can run.

Output

Experiment spec: delivery fee in basket

Owner: PM, Checkout | Readers: Ravi (build), Ines (analysis), Chloe (approval)

Hypothesis

Showing the delivery fee in the basket reduces surprise at checkout (12% of last quarter's support contacts) without reducing orders. Chloe's concern is that an early fee deters people before they start checkout, so the test measures that directly.

Changes from the team draft

DraftProblemFix
Primary metric: checkout completionIt only counts people who already started checkout. If the early fee puts people off, they never enter the denominator, so the harm would be hidden.Primary metric is orders per basket viewer.
Split by sessionReturning customers (about half of weekly viewers) would see both versions.Randomise by customer.
Check daily, stop when significantRepeated peeking inflates false positives.Fixed sample size and duration, one readout.
Add the free-delivery banner to the variantWe couldn't tell whether the fee or the banner caused any change.Banner excluded. Test it separately afterwards.

Design

  • Control: fee first shown at final checkout step (today).
  • Variant: basket shows the delivery line: £3.99, or "Free" at £60+. Nothing else changes. The basket and checkout amounts must always match.
  • Unit: customer ID, with a 50/50 split. Assignment is sticky for the whole test. Logged-out users are bucketed by a persistent device ID (Ravi to confirm how many there are).
  • Enrolment: at first basket view during the test. Log an exposure event at that moment.

Metrics

Primary: share of enrolled customers who place at least one order within 7 days of first basket view. Baseline is about 8%.

Guardrails: - Checkout start rate (baseline about 12.9%, i.e. 8% ÷ 62%). - Revenue per enrolled customer (baseline about £3.76, i.e. 8% × £47). - Delivery-fee support contacts per 1,000 customers. This is the benefit metric, but volumes are low, so read it directionally.

Diagnostics (not decision metrics): - Checkout completion (orders ÷ checkout starts). Read it only alongside the primary metric, since it's conditional on starts. - New vs returning customers, and baskets under vs over £60. Both are exploratory.

Sample size and duration

  • Smallest effect worth detecting: 0.5 percentage points on orders (8.0% → 8.5%), per Chloe.
  • Requirement: 80% power, 5% two-sided significance. This needs about 46,000 customers per arm, roughly 92,000 in total.
  • Time to reach it: Week 1 brings about 40,000 unique customers. After that, only about 20,000 new customers enrol per week, because returning customers are already assigned. That reaches about 100,000 by the end of week 4.
  • Duration: enrol for 4 full weeks, plus 7 days of follow-up for the last cohort. Readout is at about 5 weeks, with no early stopping. Ines can monitor data quality but not results.
  • Limit: the test can't reliably detect effects smaller than 0.5pp. A "no significant difference" result means "no effect larger than about 0.5pp", not "no effect".

Data quality checks (before readout)

  • Sample ratio check: arms should be within normal variation of 50/50.
  • Exposure is logged for every enrolled customer.
  • Fee shown in basket equals fee charged, in all cases.

Decision rule (Chloe to confirm)

  • Ship if the 95% confidence interval for the change in orders rules out a drop of 0.5pp or more, no guardrail is materially worse, and delivery-fee contacts do not rise.
  • Don't ship if orders fall by 0.5pp or more, or revenue per customer drops meaningfully.
  • Otherwise, treat as inconclusive and discuss. Do not extend the test to "find" significance.

Next

If the variant ships or is neutral, run a follow-up test of the free-delivery-over-£60 banner, where the fee is already visible. Its effect is likely concentrated in baskets just under £60.

Open items

  • Ravi: logged-out identity handling; a flag to switch the variant off quickly.
  • Chloe: confirm the decision rule and that the 0.5pp threshold means absolute percentage points on orders per basket viewer.
  • Ines: confirm the 7-day order attribution and the support-contact tagging for fee-related contacts.

Grades and run details

Decision model 79 · LLM judge 11 of 13 checks
Decision model checks
  • failUses the supplied evidence correctly4%
  • passAddresses the actual decision61%
  • partialRespects explicit constraints23%
  • passIdentifies material uncertainty80%
  • passAvoids unsupported claims35%
  • passProduces the required deliverable67%
  • passTests one change at a time99%
  • passFixes the duration up front100%
  • passAn unambiguous primary metric96%
  • passDecision rule written before the test95%
  • partialSized from the real traffic31%
  • partialGuardrails with thresholds94%
Run
Run
#1
API response time
29 s
Submitted
30 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 11

Uses the supplied evidence correctlyWrongWrongMixed
Gemini 3.5 Flash-Lite · Gemini

The claim that 2 weeks yields ~80,000 unique users and adequate power is not supported by the brief; the brief's traffic and returning-user dynamics do not support that sample size or power.

Gemini 3.8 Flash · API

It misstates the free-delivery threshold as ≥£60, overstates unique viewers over three weeks, and adds an unsupported >30% false-positive claim.

Sonnet 5.5 · API

All statements about the current situation are taken directly from the brief or derived by correct arithmetic.

Addresses the actual decisionMixedRightRight
Gemini 3.5 Flash-Lite · Gemini

The output commits to a specific experiment design and asks Chloe for approval, which is the deliverable requested.

Gemini 3.8 Flash · API

It commits to testing fee transparency alone, with ship, iterate, and kill rules tied to named thresholds.

Sonnet 5.5 · API

The spec commits to a clear decision rule with ship/don't ship/inconclusive actions, framed for Chloe to confirm.

Respects explicit constraintsWrongWrongRight
Gemini 3.5 Flash-Lite · Gemini

The brief says 'Fix what needs fixing' and the draft bundles the free-delivery banner; the output keeps it bundled, failing to fix that problem.

Gemini 3.8 Flash · API

It is under 700 words and addressed to the named readers, but it proposes a 21-day test that cannot reach the stated sample and includes a week-1 significance-based kill rule.

Sonnet 5.5 · API

The output is under 700 words, addresses the named readers, and is a complete experiment spec.

Identifies material uncertaintyWrongRightRight
Gemini 3.5 Flash-Lite · Gemini

The output does not name unknowns that could change the decision, nor does it specify what would happen if the result is inconclusive.

Gemini 3.8 Flash · API

It names the main uncertainty—whether early fee disclosure reduces orders—and maps outcomes to actions.

Sonnet 5.5 · API

It names unknowns like logged-out identity, decision rule confirmation, and low support-contact volumes, and says how to resolve them.

Avoids unsupported claimsWrongWrongRight
Gemini 3.5 Flash-Lite · Gemini

The output presents the 2-week duration as yielding adequate power as fact, without labelling it as an assumption, and the evidence does not support it.

Gemini 3.8 Flash · API

It presents the >30% false-positive rate and the 80,000–90,000 unique-viewer range as established facts without support.

Sonnet 5.5 · API

Hypotheses and forecasts are clearly labelled, and no interpretations are presented as established fact.

Produces the required deliverableWrongWrongRight
Gemini 3.5 Flash-Lite · Gemini

The spec is missing a decision rule, guardrail thresholds, proper sample-size justification, and a sample ratio check, so it is not usable as is.

Gemini 3.8 Flash · API

The spec is usable in form but has major gaps: the duration is underpowered, the sample-ratio check is missing, and the guardrail thresholds are incomplete.

Sonnet 5.5 · API

The spec is complete, in the right form, and usable by Ravi, Ines, and Chloe with light edits.

Tests one change at a timeWrongRightRight
Gemini 3.5 Flash-Lite · Gemini

The spec bundles the free-delivery banner into the variant, arguing it is conceptually tied, instead of testing the fee change alone.

Gemini 3.8 Flash · API

It keeps the promotional free-delivery banner out of the variant and explains that bundling it would confound attribution.

Sonnet 5.5 · API

The free-delivery banner is excluded from the variant and a separate follow-up test is proposed, with the attribution problem explained.

Fixes the duration up frontRightMixedRight
Gemini 3.5 Flash-Lite · Gemini

The spec replaces 'stop as soon as significant' with a fixed 2-week duration and explains that early peeking yields false positives.

Gemini 3.8 Flash · API

It fixes a duration and explains peeking, but the duration is not derived from the required sample and it still allows a week-1 significance-based stop.

Sonnet 5.5 · API

Duration is fixed at 4 weeks enrollment plus 7 days follow-up, with no early stopping, and the false-positive risk of peeking is explained.

An unambiguous primary metricWrongMixedRight
Gemini 3.5 Flash-Lite · Gemini

The spec names one primary metric but does not include a trust check such as a sample ratio check.

Gemini 3.8 Flash · API

It names one primary metric and rationale, but does not plan a sample-ratio check or other trustworthiness check.

Sonnet 5.5 · API

The primary metric is orders per enrolled customer within 7 days, with a clear rationale and a planned sample ratio check.

Decision rule written before the testWrongWrongRight
Gemini 3.5 Flash-Lite · Gemini

No decision rule is stated; the spec does not map outcomes (including inconclusive) to actions with thresholds.

Gemini 3.8 Flash · API

It does not define the action for an inconclusive result between the non-inferiority and harm thresholds.

Sonnet 5.5 · API

Every outcome (ship, don't ship, inconclusive) is mapped to an action with thresholds separating them.

Sized from the real trafficWrongWrongRight
Gemini 3.5 Flash-Lite · Gemini

The 2-week duration is not worked out from the traffic and MDE; the required sample size is much larger given the 8% baseline and 0.5pp effect.

Gemini 3.8 Flash · API

It calculates about 94,000 unique users but sets only three weeks, which yields about 80,000 unique users, not four weeks.

Sonnet 5.5 · API

Sample size is correctly calculated from the 8% baseline and 0.5pp effect, and duration follows from the traffic pattern over whole weeks.

All got wrong 1

Guardrails with thresholdsWrongWrongWrong
Gemini 3.5 Flash-Lite · Gemini

Guardrail metrics are listed but no thresholds are set that would block a rollout.

Gemini 3.8 Flash · API

It names guardrails but gives thresholds only for revenue, not for AOV, checkout-start rate, or support contacts.

Sonnet 5.5 · API

Guardrails are named but lack specific numeric thresholds; 'materially worse' and 'drops meaningfully' are too vague to block rollout unambiguously.

Results

Every setup we’ve tested on this task, across all cases and repeats, graded on the current checklist. Calibrated: the graders match our PM on 90% of checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6.1 SolAPI87.588.52None
2Sonnet 5.5API81.388.52None
3Opus 5.5Claude85.476.92None
4GPT-6 LunaAPI85.473.12None
5GPT-6 AstraChatGPT87.565.42None
6Gemini 3.8 FlashAPI60.438.52None
7Gemini 3.5 Flash-LiteGemini37.515.422 capped

The PM job

Specifying an A/B test before running it.

Why it matters

Most tests are underpowered, or measure a metric nobody agrees is good.

What good looks like

  • One primary metric everyone agrees on the direction of
  • Decision rule stated up front
  • Guardrails
  • Power considered

Deliberately not measured

    Capability tested

    Test design

    The failure we’re looking for

    A test with no decision attached

    Grading

    Decision model and LLM judge, calibrated against a blind PM review