Humanity’s Last PRD
Latest drop: Haiku 5.5 · 8 Oct
Tasks / Product Craft

Make the launch call

Can the model make a clear go / no-go call from mixed launch evidence, checked against the criteria agreed up front?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 69% were usable with at most a quick edit.

Reliably right

  1. Finds the Scotland breach100% pass
    Finds Scotland's refund breach (+66% overall, +143% since 15 Oct), ties it to the supplier change, notes acceptance dip, and holds Scotland.
    Sonnet 5.5 · API · Go/no-go for AI substitutions, from the rollout data
  2. Treats the overwrites as data loss100% pass
    Treats the overwritten notes on Android 11 as a breach of the no-data-loss criterion and makes fixing the storage-API issue a condition for that platform.
    Opus 5.5 · Claude · Offline, on older phones
  3. Addresses the actual decision98% pass
    Commits to a clear no-go for 2 Nov, conditional go from 4 Nov for four regions, and no-go for Scotland, with conditions that would change it.
    Sonnet 5.5 · API · Go/no-go for AI substitutions, from the rollout data

Where it slips

  1. Limits the downside of being wrong63% pass
    It does not name a specific post-launch signal, a threshold that would trigger pause or rollback, or flag any irreversible actions.
    GPT-6 Luna · API · Go/no-go for AI substitutions, from the rollout data
  2. Judges the launch against a baseline set now66% pass
    Does not name a specific review point or when the team will review results to decide next steps; only gives a pause trigger without a scheduled evaluation.
    GPT-6.1 Sol · API · Offline, on older phones
  3. Checks every criterion for every platform69% pass
    Android 13's error-free sync is 98.9%, below the 99% threshold, yet the table marks it as 'GO WITH CONDITIONS' instead of 'NO-GO' for that criterion, misapplying the hard launch criteria.
    Gemini 3.5 Flash-Lite · Gemini · Offline, on older phones

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Limits the downside of being wrong

    Does the call say how exposure is limited if it's wrong: what is watched after launch, the threshold that would pause or roll back, and which parts can't be undone?

    Passes when Names the post-launch signal, a threshold or trigger, and the action, and flags anything irreversible (customer communications, pricing, migrations).

    Karri SaarinenLenny’s Podcast

  2. Judges the launch against a baseline set now

    Does the call name the metric and the current baseline the launch will be judged against, and when the team will review the result to decide what's next?

    Passes when Names the success metric, its baseline from the material, and a review point with what each result would lead to.

    Todd JacksonLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for Lumen's mobile apps. The launch review for offline sync is on Friday, and marketing has an email booked for Monday to every user. Write your recommendation, in no more than 700 words, with a table checking each launch criterion for each platform: go, no-go or go with conditions, and why. The rollout data, the criteria and the incident log are attached.

What the model was given5 items: About Lumen, launch_criteria.md, rollout_25pct_14_days.csv, incidents.md, the_decision.md
About LumenA notes app for teams, on iOS, Android and the web. Offline sync lets people edit notes without a connection and merges changes when they reconnect.
launch_criteria.md6 lines · Download
# Launch criteria (agreed before the rollout)

1. Crash-free sessions no more than 0.2 points below control, per platform.
2. Sync-conflict tickets under 2 per 1,000 users.
3. Notes synced without error at least 99%.
4. No data loss incidents.
rollout_25pct_14_days.csv8 rows · Download
platform,arm,users,crash_free_sessions_pct,sync_conflict_tickets_per_1000_users,notes_synced_without_error_pct
iOS 17+,control,123600,99.62,0.3,99.5
iOS 17+,offline sync,41200,99.58,0.6,99.4
Android 14+,control,68400,99.55,0.3,99.5
Android 14+,offline sync,22800,99.49,0.8,99.2
Android 13,control,27300,99.4,0.3,99.5
Android 13,offline sync,9100,99.31,1.1,98.9
…
incidents.md4 lines · Download
# Incidents during the 25% rollout (14 days)

- 2 reports of a note's edits being overwritten after a conflict, both on Android 11; engineering recovered both from backups. Root cause: an older Android storage API.
- 1 report of slow sync on iOS, not reproduced.
the_decision.md1 lines · Download
The launch review is Friday: roll offline sync out to 100%, or not. Marketing has a launch email booked for Monday to every user.
What a strong answer doesThe answer key the graders mark against

Checks each criterion per platform. iOS 17+, Android 14+ and Android 13 pass: crash-free sessions within 0.2 points of control (iOS -0.04, Android 14+ -0.06, Android 13 -0.09), sync-conflict tickets under 2 per 1,000, notes synced without error at least 99%. Android 12 and older fails three criteria: crash-free down 0.67 points against control, 4.9 conflict tickets per 1,000, 96.1% synced without error; and the two overwritten-note incidents were on Android 11, which breaks 'no data loss' (both recovered from backup, root cause an older storage API). Recommends go for iOS, Android 13 and newer, and no-go for Android 12 and older until the storage issue is fixed and re-tested; the Monday email should only go to users on supported versions, or wait. States what's monitored after launch and the threshold that would pause it.

Critical failures (cap the score at 40)
  • Recommends rolling out to 100% on every platform
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Opus 5.5Claude94Usable with a quick editSonnet 5.5API94Usable with a quick editHaiku 5.5API94Usable with a quick editGPT-6.1 SolAPI90Usable with a quick editGPT-6 AstraChatGPT88Usable with a quick editGPT-6 LunaAPI75Needs more workGemini 3.8 FlashAPI71Needs more workGemini 3.5 Flash-LiteGemini52Needs more work
GPT-6.1 SolAPI

Usable with a quick edit10 of 11 checks passed

The output is clear, evidence is correctly used, and a PM could act on it with minor edits; no re-checking of facts or reasoning is needed.

Output

Recommendation for Friday

No-go for a blanket rollout to 100% of users. Approve expansion on iOS 17+ and Android 14+ only, with the safeguards below. Hold Android 13 and Android 12 and older. Do not approve web without launch evidence.

The agreed criteria are launch gates, not averages across platforms. Strong iOS results cannot offset failures on older Android versions.

Launch-gate assessment

“Go” below means the supplied evidence meets that criterion; it is not an unconditional platform launch approval.

PlatformCrash-free sessions: ≤0.2 percentage-point declineConflict tickets: <2/1,000Successful sync: ≥99%No data lossPlatform decision
iOS 17+Go: 99.58%; down 0.04 points vs. controlGo: 0.6/1,000Go: 99.4%Go: no reported incidentsGo with conditions: meets all gates; use monitored expansion
Android 14+Go: 99.49%; down 0.06 pointsGo: 0.8/1,000Go: 99.2%Go with conditions: no reported incidents; confirm the older-storage-API defect cannot affect this cohortGo with conditions: expand only after that confirmation
Android 13Go: 99.31%; down 0.09 pointsGo: 1.1/1,000No-go: 98.9%, below 99%Go: no reported incidentsNo-go: fails successful-sync gate
Android 12 and olderNo-go: 98.71%; down 0.67 pointsNo-go: 4.9/1,000No-go: 96.1%No-go: two overwritten-edit incidents on Android 11No-go: fails all four gates
WebNo-go: no evidence suppliedNo-go: no evidence suppliedNo-go: no evidence suppliedNo-go: absence of incidents is not establishedNo-go for approval: insufficient evidence, not a demonstrated product failure

Conditions and actions

  • Treat the overwritten edits as data-loss incidents. Backup recovery mitigated their impact; it does not satisfy “no data loss incidents.” Engineering’s root-cause finding is not evidence that a fix has shipped or passed validation.
  • Before expanding eligible cohorts, confirm platform targeting excludes held cohorts, establish monitoring and a tested kill switch, and confirm the Android storage defect is isolated. Expand in monitored steps toward 100% of eligible users. Pause if any agreed gate fails or a data-loss incident occurs.
  • Stop further exposure on Android 12 and older. Engineering should determine the safest disablement path for existing participants without stranding unsynced edits. Require a validated fix and fresh rollout evidence meeting all four gates before reconsideration.
  • Keep Android 13 out of general launch. Its 98.9% result is close, but still fails the pre-agreed threshold. Investigate sync errors and rerun validation; do not lower the gate to meet the marketing date.
  • Investigate the unreproduced iOS slow-sync report, but do not treat it as a failed gate on current evidence.

Monday’s email

Do not send the booked launch email to every user. Retarget it to approved, eligible users only, and send only if Friday’s conditions are satisfied and expansion remains healthy. State supported OS versions clearly; do not imply availability on web or held Android versions.

If segmentation or readiness cannot be confirmed, postpone the email. Marketing timing should follow launch readiness—not override the agreed reliability and data-integrity gates.

Grades and run details

Decision model 86 · LLM judge 92
Decision model checks
  • passUses the supplied evidence correctly51%
  • passAddresses the actual decision85%
  • partialRespects explicit constraints22%
  • passIdentifies material uncertainty95%
  • passAvoids unsupported claims34%
  • passProduces the required deliverable74%
  • partialChecks every criterion for every platform60%
  • passTreats the overwrites as data loss100%
  • passLimits the downside of being wrong49%
  • passGets the base of every number right73%
  • partialJudges the launch against a baseline set now29%
Run
Run
#1
API response time
27 s
Submitted
6 Oct 2026
Sonnet 5.5API

Usable with a quick edit11 of 11 checks passed

The output is factually correct, well-structured, and includes all required elements; a PM could act on it with at most minor wording tweaks.

Output

Recommendation: partial go

Ship offline sync to 100% on iOS 17+ and Android 14+. Do not ship it on Android 13 or Android 12 and older. Segment Monday's email so it doesn't promise the feature to users who won't have it.

Criteria check (25% rollout, 14 days)

Platform1. Crash-free (≥ control −0.2)2. Conflict tickets (<2/1,000)3. Synced w/o error (≥99%)4. No data lossVerdict
iOS 17+Go: −0.04Go: 0.6Go: 99.4%Go: none (one slow-sync report, not reproduced)Go
Android 14+Go: −0.06Go: 0.8Go: 99.2%Go: noneGo
Android 13Go: −0.09Go: 1.1, but over half the limitNo-go: 98.9%, misses by 0.1Go: none reportedNo-go for now
Android 12 and olderNo-go: −0.67No-go: 4.9, about 2.5× the limitNo-go: 96.1%No-go: 2 overwrites on Android 11No-go

Reasoning

iOS 17+ and Android 14+ pass every criterion with margin. Together they are about 80% of users. The iOS slow-sync report wasn't reproduced, so I'd watch it during the ramp but not block on it.

Android 13 misses one criterion by 0.1 points. That is small, but we agreed the criteria before seeing the data, and I don't want to bend them the first time a result is inconvenient. The trend also worries me. Sync success falls with each older Android tier (99.2 → 98.9 → 96.1), and conflict tickets rise (0.8 → 1.1 → 4.9). This looks like a gradient tied to the OS, not noise, so Android 13 sits on the slope. I'd hold it at its current exposure, find out why it's degrading, and re-measure. If it reaches 99% or better with conflicts still under 2, it can go.

Android 12 and older fails all four criteria, and it is not close. Crash-free sessions are 0.67 points below control, more than three times the allowed gap. Conflict tickets are about 2.5× the limit, and sync success is 3 points under the bar. The two data-loss incidents were edits overwritten after a conflict, on Android 11. Engineering recovered both from backups, but that doesn't make them acceptable. We only recovered them because users reported them, and silent overwrites may not have been reported. The root cause is an older Android storage API, so this is a structural problem and not a quick patch. I'd also disable offline sync for the users currently exposed on Android 11 and below until it's fixed. Data loss in a notes app is the failure users won't forgive.

The Monday email

An email to every user would promise a feature to roughly 20% of users (Android 13 and older, about 66,000 of 322,000) who won't have it. That means confusion, support tickets, and a bad first impression of the feature. My recommendation is:

  1. Send Monday to iOS 17+ and Android 14+ users only, and only if the ramp over the weekend stays within the criteria. Marketing should confirm it can segment by platform and OS version.
  2. Hold the email for Android 13 and older. Send it when each segment passes, or send a "coming to your device soon" version if marketing wants one.
  3. Check the ramp Monday morning. If crash-free sessions or sync success slip on either approved platform, pause the email and the ramp.

Conditions and next steps

  • Android 12 and older: Engineering should scope a fix for the storage API issue, and we should re-test in a new, smaller cohort. I'd also like the data split into Android 12 versus 11 and below. The incidents were all on Android 11, so Android 12 itself may be fine, but this bucket can't tell us.
  • Android 13: Investigate the sync errors and re-measure in 1–2 weeks.
  • Data-loss monitoring: Add a check for overwrite after conflict, so we aren't relying on users to tell us.

Caveats

  • The data covers 14 days at 25%. The Android 13 miss is small enough that more data could move it either way.
  • Per-platform samples are decent (7,400 to 41,200 users), but two data-loss incidents is a tiny count. Treat it as a signal and not an estimate of the true rate.

Grades and run details

Decision model 86 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly33%
  • passAddresses the actual decision85%
  • partialRespects explicit constraints24%
  • passIdentifies material uncertainty97%
  • partialAvoids unsupported claims28%
  • passProduces the required deliverable72%
  • partialChecks every criterion for every platform49%
  • passTreats the overwrites as data loss100%
  • passLimits the downside of being wrong77%
  • passGets the base of every number right29%
  • passJudges the launch against a baseline set now85%
Run
Run
#1
API response time
28 s
Submitted
6 Oct 2026
GPT-6 LunaAPI

Needs more than a quick edit9 of 11 checks passed

Missing a post-launch monitoring plan with thresholds and a baseline for success, which a PM would need to add before acting.

Output

Recommendation

Do not roll offline sync out to every user or send Monday’s email to every user as a general launch announcement. The agreed criteria are not met on Android 13 or Android 12 and older, and the Android 11 overwrite reports fail the no-data-loss gate even though the notes were recovered from backups.

Approve rollout to 100% only for iOS 17+ and Android 14+, where all four criteria pass. Keep Android 13 and older gated until they meet the criteria and the Android 11 storage issue is fixed and validated. Change Monday’s email to target eligible users, or postpone it; don’t imply the feature is available to everyone. Monitor the unreproduced iOS slow-sync report.

Platform / cohortDecisionCrash-free sessionsConflict ticketsNotes synced without errorNo data loss
iOS 17+GoGo — 0.04 percentage points below control; within 0.2.Go — 0.6/1,000; under 2.Go — 99.4%; at least 99%.Go — no data-loss incident reported.
Android 14+GoGo — 0.06 points below control; within 0.2.Go — 0.8/1,000; under 2.Go — 99.2%; at least 99%.Go — no data-loss incident reported.
Android 13Go with conditions — hold rollout until notes-sync criterion passes.Go — 0.09 points below control; within 0.2.Go — 1.1/1,000; under 2.No-go — 98.9%; below 99%.Go — no data-loss incident reported.
Android 12 and olderNo-go — keep blocked; investigate and validate by OS version.No-go — 0.67 points below control; exceeds 0.2.No-go — 4.9/1,000; exceeds 2.No-go — 96.1%; below 99%.No-go — the cohort includes Android 11, where two overwritten edits were reported. Recovery does not erase the incident.

The Android 12-and-older metrics are grouped, so they do not establish that each OS version performs equally; assess Android 12 separately before reconsidering it.

Grades and run details

Decision model 82 · LLM judge 75
Decision model checks
  • passUses the supplied evidence correctly67%
  • passAddresses the actual decision79%
  • passRespects explicit constraints34%
  • partialIdentifies material uncertainty36%
  • passAvoids unsupported claims22%
  • passProduces the required deliverable43%
  • partialChecks every criterion for every platform45%
  • passTreats the overwrites as data loss100%
  • partialLimits the downside of being wrong98%
  • passGets the base of every number right66%
  • partialJudges the launch against a baseline set now82%
Run
Run
#1
API response time
21 s
Submitted
6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 2

Limits the downside of being wrongRightRightWrong
GPT-6.1 Sol · API

Specifies monitoring, a kill switch, and a pause trigger (any gate failure or data-loss incident), and addresses irreversible exposure on Android 12 and older.

Sonnet 5.5 · API

Specifies post-launch monitoring (crash-free, sync success), a trigger (slip), and action (pause email and ramp), and flags the irreversible email communication.

GPT-6 Luna · API

Does not name a post-launch signal, threshold, or action to pause or roll back for the platforms going live, nor flag anything irreversible.

Judges the launch against a baseline set nowWrongRightWrong
GPT-6.1 Sol · API

Does not name a specific review point or when the team will review results to decide next steps; only gives a pause trigger without a scheduled evaluation.

Sonnet 5.5 · API

Names the success metrics (crash-free sessions, sync success), implies the baseline from the go criteria, and sets a review point (Monday morning) with a clear action if they slip.

GPT-6 Luna · API

Does not name a success metric, its current baseline, or a review point with what each result would lead to.

All got right 9

Uses the supplied evidence correctlyRightRightRight
GPT-6.1 Sol · API

All facts and figures come directly from the supplied rollout data and incident log, with no invented information.

Sonnet 5.5 · API

All factual claims about the current situation are directly supported by the supplied data or simple arithmetic from it.

GPT-6 Luna · API

All facts and figures are taken correctly from the supplied context, with no invented information.

Addresses the actual decisionRightRightRight
GPT-6.1 Sol · API

Commits clearly to a no-go for blanket rollout, approves iOS 17+ and Android 14+ only, and states conditions that would change the call.

Sonnet 5.5 · API

Commits early to a clear partial-go recommendation (ship to iOS 17+ and Android 14+, hold Android 13 and older) and states what would change each call.

GPT-6 Luna · API

Commits to a clear answer (rollout only to iOS 17+ and Android 14+, gate others, change email) and states conditions that would change it.

Respects explicit constraintsRightRightRight
GPT-6.1 Sol · API

Stays within 700 words, includes the required table with go/no-go/conditions per platform, and addresses the launch review audience.

Sonnet 5.5 · API

Output is under 700 words, includes the required table with go/no-go per platform, and addresses the launch review decision and Monday email.

GPT-6 Luna · API

Respects the 700-word limit, includes the required table with go/no-go per platform and criterion, and addresses the Monday email.

Identifies material uncertaintyRightRightRight
GPT-6.1 Sol · API

Identifies the storage API defect isolation, the unreproduced iOS slow sync, and segmentation readiness as unknowns, and says how to resolve them.

Sonnet 5.5 · API

Names the small Android 13 miss, the need to split Android 12 vs 11, and the tiny data-loss count as uncertainties, and says how to resolve them.

GPT-6 Luna · API

Names the grouped Android 12-and-older data as an unknown and says to assess Android 12 separately; also flags the unreproduced iOS slow-sync report to monitor.

Avoids unsupported claimsRightRightRight
GPT-6.1 Sol · API

Presents the overwritten edits as data-loss incidents as a judgment call, not an established fact, and labels conditions clearly.

Sonnet 5.5 · API

Interpretations like the OS gradient are presented as hypotheses ('looks like'), and no unsupported facts are stated as certain.

GPT-6 Luna · API

No interpretations or forecasts are presented as established fact; the recovery-not-erasing-incident statement is a correct application of the criterion.

Produces the required deliverableRightRightRight
GPT-6.1 Sol · API

Provides a complete recommendation with the required table, within the word limit, and is directly usable by the launch review audience.

Sonnet 5.5 · API

Provides a complete recommendation with table, reasoning, email plan, and next steps, usable by a product manager with light edits.

GPT-6 Luna · API

The recommendation is complete, in the right form for a launch review, within the word limit, and usable with light edits.

Checks every criterion for every platformRightRightRight
GPT-6.1 Sol · API

Table correctly computes each criterion for every platform, and correctly identifies Android 12 and older as failing.

Sonnet 5.5 · API

Table correctly computes each criterion for every platform against control and thresholds, and correctly identifies Android 12 and older as failing.

GPT-6 Luna · API

The table correctly computes each criterion for each platform against control and thresholds, and correctly identifies Android 12 and older as failing.

Treats the overwrites as data lossRightRightRight
GPT-6.1 Sol · API

Explicitly treats the Android 11 overwritten edits as a breach of the no-data-loss criterion and makes a fix a condition for that platform.

Sonnet 5.5 · API

Treats the Android 11 overwrites as a breach of the no-data-loss criterion and makes a fix a condition for that platform.

GPT-6 Luna · API

Treats the Android 11 overwrites as a breach of the no-data-loss criterion and makes fixing the storage issue a condition for that platform.

Gets the base of every number rightRightRightRight
GPT-6.1 Sol · API

All percentage differences are computed correctly from the control and offline-sync values, and the bases (per 1,000 users, percentage points) are clear.

Sonnet 5.5 · API

All derived figures (differences, percentages, user shares) are computed correctly from the supplied data, with no ambiguous bases.

GPT-6 Luna · API

All derived differences and percentages are computed correctly from the supplied data, and bases are clear.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 63% of 68 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Sonnet 5.5API87.8100.02None
2Haiku 5.5API82.5100.02None
3GPT-6.1 SolAPI86.095.82None
4Opus 5.5Claude84.396.72None
5GPT-6 AstraChatGPT83.892.52None
6GPT-6 LunaAPI74.880.82None
7Gemini 3.8 FlashAPI64.967.521 capped
8Gemini 3.5 Flash-LiteGemini48.259.22None

What you get for the cost

Haiku 5.5 · API scores within 2 points of Sonnet 5.5 · API at about a tenth of the cost.

Task score against relative cost for each setup. The table below has the same figures.4050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreSonnet 5.5 · API: task score 94, 37× the cheapestHaiku 5.5 · API: task score 92, 2.8× the cheapestGPT-6.1 Sol · API: task score 91, 12× the cheapestOpus 5.5 · Claude: task score 91, 59× the cheapest (1 of 2 outputs estimated)GPT-6 Astra · ChatGPT: task score 89, 46× the cheapest (1 of 2 outputs estimated)GPT-6 Luna · API: task score 77, the cheapestGemini 3.8 Flash · API: task score 55, 11× the cheapest, 1 critical failureGemini 3.5 Flash-Lite · Gemini: task score 52, 2.7× the cheapest (1 of 2 outputs estimated)Sonnet 5.5 · APIHaiku 5.5 · APIGPT-6.1 Sol · APIOpus 5.5 · ClaudeGPT-6 Astra · ChatGPTGPT-6 Luna · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)At least one critical failure

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Deciding whether a feature ships on the planned date, and on what conditions.

Why it matters

Launch meetings reward optimism. A good call checks each agreed criterion, weighs a real risk against a real gain, and says exactly what would change the answer. A weak one rubber-stamps the launch or blocks it on noise.

What good looks like

  • Checks each agreed criterion against the evidence
  • Makes one clear call, with conditions if needed
  • Separates risks that block launch from risks that can be managed
  • Says what would change the call
  • Says how to limit the damage if the call is wrong

Deliberately not measured

  • Rollout engineering detail
  • Project-plan formatting
Capability tested

Deciding against agreed launch criteria

The failure we’re looking for

Rubber-stamps a launch that misses an agreed bar, or blocks it on noise

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

A launch memo from supplied evidence · Staff level: a regional call from an attached data workbook