Tasks / Product Craft

Make the launch call

Can the model make a clear go / no-go call from mixed launch evidence, checked against the criteria agreed up front?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 69% were usable with at most a quick edit.

Reliably right

Finds the Scotland breach100% pass
Finds Scotland's refund breach (+66% overall, +143% since 15 Oct), ties it to the supplier change, notes acceptance dip, and holds Scotland.
Sonnet 5.5 · API · Go/no-go for AI substitutions, from the rollout data
Treats the overwrites as data loss100% pass
Treats the overwritten notes on Android 11 as a breach of the no-data-loss criterion and makes fixing the storage-API issue a condition for that platform.
Opus 5.5 · Claude · Offline, on older phones
Addresses the actual decision98% pass
Commits to a clear no-go for 2 Nov, conditional go from 4 Nov for four regions, and no-go for Scotland, with conditions that would change it.
Sonnet 5.5 · API · Go/no-go for AI substitutions, from the rollout data

Where it slips

Limits the downside of being wrong63% pass
It does not name a specific post-launch signal, a threshold that would trigger pause or rollback, or flag any irreversible actions.
GPT-6 Luna · API · Go/no-go for AI substitutions, from the rollout data
Judges the launch against a baseline set now66% pass
Does not name a specific review point or when the team will review results to decide next steps; only gives a pause trigger without a scheduled evaluation.
GPT-6.1 Sol · API · Offline, on older phones
Checks every criterion for every platform69% pass
Android 13's error-free sync is 98.9%, below the 99% threshold, yet the table marks it as 'GO WITH CONDITIONS' instead of 'NO-GO' for that criterion, misapplying the hard launch criteria.
Gemini 3.5 Flash-Lite · Gemini · Offline, on older phones

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

Limits the downside of being wrong
Does the call say how exposure is limited if it's wrong: what is watched after launch, the threshold that would pause or roll back, and which parts can't be undone?
Passes when Names the post-launch signal, a threshold or trigger, and the action, and flags anything irreversible (customer communications, pricing, migrations).
Karri SaarinenLenny’s Podcast
Judges the launch against a baseline set now
Does the call name the metric and the current baseline the launch will be judged against, and when the team will review the result to decide what's next?
Passes when Names the success metric, its baseline from the material, and a review point with what each result would lead to.
Todd JacksonLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for Lumen's mobile apps. The launch review for offline sync is on Friday, and marketing has an email booked for Monday to every user. Write your recommendation, in no more than 700 words, with a table checking each launch criterion for each platform: go, no-go or go with conditions, and why. The rollout data, the criteria and the incident log are attached.

What the model was given5 items: About Lumen, launch_criteria.md, rollout_25pct_14_days.csv, incidents.md, the_decision.md

About LumenA notes app for teams, on iOS, Android and the web. Offline sync lets people edit notes without a connection and merges changes when they reconnect.

launch_criteria.md6 lines · Download

# Launch criteria (agreed before the rollout)

1. Crash-free sessions no more than 0.2 points below control, per platform.
2. Sync-conflict tickets under 2 per 1,000 users.
3. Notes synced without error at least 99%.
4. No data loss incidents.

rollout_25pct_14_days.csv8 rows · Download

platform,arm,users,crash_free_sessions_pct,sync_conflict_tickets_per_1000_users,notes_synced_without_error_pct
iOS 17+,control,123600,99.62,0.3,99.5
iOS 17+,offline sync,41200,99.58,0.6,99.4
Android 14+,control,68400,99.55,0.3,99.5
Android 14+,offline sync,22800,99.49,0.8,99.2
Android 13,control,27300,99.4,0.3,99.5
Android 13,offline sync,9100,99.31,1.1,98.9
…

incidents.md4 lines · Download

# Incidents during the 25% rollout (14 days)

- 2 reports of a note's edits being overwritten after a conflict, both on Android 11; engineering recovered both from backups. Root cause: an older Android storage API.
- 1 report of slow sync on iOS, not reproduced.

the_decision.md1 lines · Download

The launch review is Friday: roll offline sync out to 100%, or not. Marketing has a launch email booked for Monday to every user.

What a strong answer doesThe answer key the graders mark against

Checks each criterion per platform. iOS 17+, Android 14+ and Android 13 pass: crash-free sessions within 0.2 points of control (iOS -0.04, Android 14+ -0.06, Android 13 -0.09), sync-conflict tickets under 2 per 1,000, notes synced without error at least 99%. Android 12 and older fails three criteria: crash-free down 0.67 points against control, 4.9 conflict tickets per 1,000, 96.1% synced without error; and the two overwritten-note incidents were on Android 11, which breaks 'no data loss' (both recovered from backup, root cause an older storage API). Recommends go for iOS, Android 13 and newer, and no-go for Android 12 and older until the storage issue is fixed and re-tested; the Monday email should only go to users on supported versions, or wait. States what's monitored after launch and the threshold that would pause it.

Critical failures (cap the score at 40)

Recommends rolling out to 100% on every platform

Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Opus 5.5Claude94Usable with a quick edit Sonnet 5.5API94Usable with a quick edit Haiku 5.5API94Usable with a quick edit GPT-6.1 SolAPI90Usable with a quick edit GPT-6 AstraChatGPT88Usable with a quick edit GPT-6 LunaAPI75Needs more work Gemini 3.8 FlashAPI71Needs more work Gemini 3.5 Flash-LiteGemini52Needs more work

Haiku 5.5API

Usable with a quick edit11 of 11 checks passed

The output is clear, actionable, and covers all required elements. Minor clarification of the Android 14+ conditions could be added without any rework of evidence or logic.

Output

Recommendation: offline sync to iOS 17+ and Android 14+ only

Don't roll out to 100% globally on Friday. Roll it out to iOS 17+ and Android 14+, hold Android 13, and turn it off for Android 12 and older. Marketing's Monday email should go only to the users who get the feature.

Platform	1. Crash-free (≤0.2 below control)	2. Conflict tickets (<2 per 1,000)	3. Notes synced without error (≥99%)	4. No data loss	Decision and why
iOS 17+	Go: 99.58 vs 99.62 (−0.04)	Go: 0.6 (control 0.3)	Go: 99.4 (control 99.5)	Go: no data-loss reports	Go. Passes all four. The one slow-sync report wasn't reproduced, so I'd watch it but it doesn't block launch.
Android 14+	Go: 99.49 vs 99.55 (−0.06)	Go: 0.8 (control 0.3)	Go: 99.2 (control 99.5)	Go: none logged	Go with conditions. Passes all four, but conflicts are 2.7× control and sync errors are slightly worse. Conditions below.
Android 13	Go: 99.31 vs 99.40 (−0.09)	Go: 1.1 (control 0.3)	No-go: 98.9, below 99%	Go: none logged	No-go. Fails the sync-error criterion. Reopen when it reaches 99% in a new test.
Android 12 and older	No-go: 98.71 vs 99.38 (−0.67)	No-go: 4.9 (control 0.3)	No-go: 96.1	No-go: two overwritten-edit incidents on Android 11	No-go. Fails all four criteria. The data-loss incidents are the decisive problem.

Web isn't in the 25% rollout data, so there's no evidence it meets any criterion. It's a no-go until it's tested.

On the data-loss criterion: the two Android 11 incidents count as failures even though engineering recovered the edits from backups. The criterion is "no data loss incidents," not "no unrecoverable data loss." Their root cause is an older Android storage API, which suggests the problem is specific to older devices. The Android 12-and-older segment also fails the crash and conflict criteria by a wide margin, so the storage fix alone doesn't justify launching there.

Why not a global launch

A global launch would put Android 12-and-older and Android 13 users on a feature that fails the agreed criteria. That segment already shows about 36 conflict tickets per 1,000 treated users... no, 4.9 per 1,000, almost 16 times control, and a 0.67-point crash-free drop. Data loss is the most serious harm a notes app can cause, and users who lose work are unlikely to forgive it.

We agreed these criteria before the rollout. Passing three of four on the platforms that matter most is not enough to waive the fourth, and waiving it now would set a bad precedent.

Before Friday

1. Engineering: Confirm the Android 11 storage fix and check that no other users had edits overwritten. Verify the backup recovery was complete. 2. Feature flag: Build a server-side switch that disables offline sync for Android 13 and below. Before disabling it on a device, flush any pending offline edits. Otherwise a user could lose changes made offline during the switch-off. 3. Monitoring: Set daily alerts with rollback triggers. Roll back any enabled platform if any of these happen: - Any data-loss report - Crash-free sessions more than 0.2 below control - Conflict tickets at or above 2 per 1,000 - Android 14+ conflicts above 1.5 per 1,000 (an early warning, since it's already at 0.8)

Monday's email

Don't send it to every user. Send it only to iOS 17+ and Android 14+ users, and only after the feature is live for them. The email should not promise offline editing on web or on older Android versions. If marketing can't segment the list in time, move the email back a week. A broad email would drive users on unsupported platforms to a feature that is currently losing their notes.

Summary

Launch Friday: iOS 17+ (go), Android 14+ (go with conditions)
Hold: Android 13 (no-go until sync errors reach 99%)
Turn off now: Android 12 and older (no-go)
Web: no-go until tested
Email: segmented to launched platforms only

Claims the judge couldn’t find in the brief

The two Android 11 incidents breach the 'no data loss' criterion, even though edits were recovered.

Grades and run details

Decision model 86 · LLM judge 100

Decision model checks

passUses the supplied evidence correctly33%
passAddresses the actual decision94%
partialRespects explicit constraints35%
passIdentifies material uncertainty97%
partialAvoids unsupported claims34%
passProduces the required deliverable82%
partialChecks every criterion for every platform62%
passTreats the overwrites as data loss100%
passLimits the downside of being wrong100%
passGets the base of every number right28%
passJudges the launch against a baseline set now58%

Run

Run: #1
API response time: 26 s
Submitted: 8 Oct 2026

Gemini 3.5 Flash-LiteGemini

Needs more than a quick edit5 of 11 checks passed

The output incorrectly marks Android 13 as go with conditions despite failing the error-free sync criterion, and lacks post-launch monitoring thresholds, so a PM would need to re-check the criteria and add missing guardrails.

Output

To: Leadership Team From: PM, Lumen Mobile Apps Date: March 28, 2024 Subject: Recommendation: Offline Sync 100% Rollout (Friday Launch Review)

---

Executive Summary

We have completed the 14-day, 25% rollout of Offline Sync across iOS and Android. Based on the data, incident logs, and agreed launch criteria, my recommendation is a segmented decision for Monday’s full rollout:

iOS 17+: GO. All metrics meet or exceed criteria with zero critical incidents.
Android 14+ and Android 13: GO WITH CONDITIONS. Stability is acceptable, but minor degradation requires a hotfix commitment before 100% exposure.
Android 12 and older: NO-GO. Fails multiple hard launch criteria, including data loss incidents. We must exclude these OS versions from the 100% rollout.

---

Launch Criteria Status Matrix

Platform / OS	1. Crash-Free (Max -0.2% vs control)	2. Conflict Tickets (<2 / 1k users)	3. Error-Free Sync (≥99%)	4. No Data Loss	Final Recommendation & Rationale
:---	:---:	:---:	:---:	:---:	:---
iOS 17+	GO<br>(99.58% vs 99.62% control; Δ -0.04)	GO<br>(0.6 / 1k)	GO<br>(99.4%)	GO<br>(0 incidents)	GO. Performs well within tolerance. One unverified slow-sync report does not block.
Android 14+	GO<br>(99.49% vs 99.55% control; Δ -0.06)	GO<br>(0.8 / 1k)	GO<br>(99.2%)	GO<br>(0 incidents)	GO WITH CONDITIONS. Stable, but error-free sync is close to the 99% floor. Ship with expedited telemetry monitoring.
Android 13	GO<br>(99.31% vs 99.40% control; Δ -0.09)	GO<br>(1.1 / 1k)	GO WITH CONDITIONS<br>(98.9%)	GO<br>(0 incidents)	GO WITH CONDITIONS. Sync error rate dips slightly below 99% (98.9%). Acceptable for rollout only alongside a targeted patch.
Android 12 and older	NO-GO<br>(98.71% vs 99.38% control; Δ -0.67)	NO-GO<br>(4.9 / 1k)	NO-GO<br>(96.1%)	NO-GO<br>(2 data loss incidents)	NO-GO. Fails all criteria. Legacy storage APIs caused two overwrites. Exclude from 100% release.

---

Key Findings & Rationale

iOS is Fully Vetted: iOS users experience virtually identical stability to the control group. The single support ticket regarding slow sync was an isolated, non-reproducible event.
Modern Android is Stable, but Needs Polish: Android 13 and 14+ show minor dips in error-free sync rates (98.9% and 99.2%, respectively) and slight increases in conflict tickets, but both remain safely within agreed thresholds and exhibit zero data loss.
Legacy Android is a Risk: Android 12 and older failed Criterion 1 (crash-free sessions dropped by 0.67%), Criterion 2 (4.9 conflict tickets per 1k users), Criterion 3 (96.1% sync success), and violated Criterion 4 (two confirmed data loss incidents due to legacy storage API conflicts). Although engineering successfully recovered the notes from backups, data loss is a hard stop for a 100% rollout.

---

Action Plan for Monday’s Launch

1. Modify the Rollout Scope: * Push 100% of Offline Sync to iOS 17+ and Android 13+. * Halt the rollout for Android 12 and older, keeping these users on the standard online-only experience until a refactored storage handler is built. 2. Marketing Alignment: Proceed with Monday’s email blast to all users, with a minor internal note to customer support regarding known edge cases on legacy Android devices. 3. Engineering Post-Launch Focus: Prioritize a patch for legacy Android storage APIs to bring older devices up to launch standard for a secondary release next quarter.

Grades and run details

Decision model 50 · LLM judge 58

Decision model checks

failUses the supplied evidence correctly16%
partialAddresses the actual decision37%
failRespects explicit constraints35%
partialIdentifies material uncertainty87%
partialAvoids unsupported claims30%
partialProduces the required deliverable44%
partialChecks every criterion for every platform25%
passTreats the overwrites as data loss96%
partialLimits the downside of being wrong96%
passGets the base of every number right49%
partialJudges the launch against a baseline set now82%

Run

Run: #1
Time to output: 5 s
Submitted: 5 Oct 2026

Opus 5.5Claude

Usable with a quick edit11 of 11 checks passed

The output is clear, data-backed, and includes all required elements; a PM could use it with minimal edits, without rechecking evidence or redoing reasoning.

Output

Offline sync launch recommendation

Recommendation: Go with conditions on Friday, for iOS 17+ and Android 14+ only. Hold Android 13. Turn the feature off for Android 12 and older. Do not send Monday's email to every user.

Two segments meet every criterion: iOS 17+ and Android 14+. Android 13 fails the sync-reliability bar. Android 12 and older fails three of four criteria and is where both data-loss incidents occurred. The criteria were agreed before the rollout, so I don't recommend relaxing them now that we have results.

Criteria check (25% rollout, 14 days)

Platform	1. Crash-free gap vs control (max 0.2 pts)	2. Conflict tickets (under 2 per 1,000)	3. Notes synced without error (≥99%)	4. No data loss	Decision
iOS 17+	✅ 0.04 pts (99.58 vs 99.62)	✅ 0.6	✅ 99.4%	✅ None. One slow-sync report, not reproduced	Go
Android 14+	✅ 0.06 pts (99.49 vs 99.55)	✅ 0.8	✅ 99.2%	✅ None reported	Go with conditions: engineering confirms the Android 11 storage-API bug can't occur on 14+
Android 13	✅ 0.09 pts (99.31 vs 99.40)	✅ 1.1	❌ 98.9%	✅ None reported	No-go: misses the agreed sync bar. Hold at 25% and fix
Android 12 and older	❌ 0.67 pts (98.71 vs 99.38)	❌ 4.9, about 16× control	❌ 96.1%	❌ 2 incidents (Android 11), recovered only from backups	No-go: turn off for current rollout users
Web	No data	No data	No data	No data	Not assessable: wasn't in the rollout

Why

iOS 17+ and Android 14+ are clean. Every metric clears its threshold with margin. The crash gaps are a fraction of the 0.2-point allowance. Conflict tickets are about 2–3× control, which is expected when people can now edit offline, and still well under the limit. The only condition on Android 14+ is confirmation that the overwrite bug is tied to the older storage API and can't occur there.

Android 13 is a near miss, but still a miss. It passes three criteria. Its 98.9% sync rate is just under the 99% bar. It is also trending worse than 14+ on every metric. Shipping it to 100% would mean overriding a criterion we agreed in advance. I'd rather fix the cause and re-review it in a follow-up launch.

Android 12 and older is unsafe. Roughly 1 in 25 notes fails to sync. The crash gap is more than three times the allowance. Two users had edits overwritten. We recovered those edits only because engineering pulled them from backups, which doesn't scale, and a user who never reports the problem simply loses their work. I recommend turning offline sync off now for the 7,400 users on this segment, not just excluding them from 100%.

Gaps in coverage: - Web has no rollout data at all. - iOS below 17 wasn't in the rollout, so it is untested.

Both should stay off until they have their own rollout.

Conditions for Friday's go

OS-version targeting. The feature flag can target by platform and OS version, so 100% means 100% of iOS 17+ and Android 14+ only. If it can't, the answer for Friday is no-go.
Engineering sign-off on Android 14+. Written confirmation that the overwrite root cause doesn't apply.
Off for Android 12 and older now. Done before or at launch, with a check that no user loses offline edits made while it was on.
Monitoring after launch. We watch crash-free sessions, conflict tickets and sync errors daily for two weeks. We roll back if any segment crosses a threshold.

Marketing email

An email to every user on Monday would promote offline sync to people who can't get it, including all web users, older iOS and Android 13 and below. For the oldest Android group, it would point them at the version we know loses data. My options, in order of preference:

Segment the email to iOS 17+ and Android 14+ users, if marketing can target by OS version.
Postpone the email until we can segment it, or until more platforms qualify.
Change the copy to say "now on iOS and newer Android, more platforms coming." This is a last resort, because it still lands with users who can't use the feature.

Next steps

Android 13: engineering investigates the sync-error rate, then we re-review.
Android 12 and older: fix the storage-API issue and run a fresh rollout before it's considered again.
Web and iOS below 17: start their own 25% rollouts against the same criteria.

Grades and run details

Decision model 86 · LLM judge 100

Decision model checks

passUses the supplied evidence correctly32%
passAddresses the actual decision91%
partialRespects explicit constraints35%
passIdentifies material uncertainty97%
partialAvoids unsupported claims48%
passProduces the required deliverable86%
partialChecks every criterion for every platform47%
passTreats the overwrites as data loss100%
passLimits the downside of being wrong57%
passGets the base of every number right51%
passJudges the launch against a baseline set now84%

Run

Run: #1
Time to output: 30 s
Submitted: 5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 6

Uses the supplied evidence correctlyRightMixedRight

Haiku 5.5 · API

All numerical facts and incident details are taken correctly from the supplied rollout data and incident log; the interpretation of the data-loss criterion is a judgment but does not invent new facts.

Gemini 3.5 Flash-Lite · Gemini

All factual claims about current performance, numbers, and incidents are taken directly from the supplied rollout data and incident log, with no invented facts.

Opus 5.5 · Claude

All factual claims about the current situation are taken directly from the supplied rollout data and incident log, with no invented facts.

Respects explicit constraintsRightMixedRight

Haiku 5.5 · API

Stays under 700 words, includes the required table, and addresses both the Friday launch decision and the Monday email.

Gemini 3.5 Flash-Lite · Gemini

The output is a recommendation memo under 700 words, includes the required table with go/no-go/go with conditions per platform, and addresses the leadership team as the reader.

Opus 5.5 · Claude

The output is under 700 words, includes the required table with go/no-go/go with conditions per platform, and is written as a recommendation for the PM.

Identifies material uncertaintyRightWrongRight

Haiku 5.5 · API

Calls out the unreproduced slow-sync on iOS, absent web data, and need for engineering confirmation of the storage fix; ties them to monitoring thresholds.

Gemini 3.5 Flash-Lite · Gemini

Does not name specific unknowns that could change the decision, does not bound them, and does not say how they would be resolved or what post-launch signal would trigger a pause or rollback.

Opus 5.5 · Claude

Identifies gaps (web and older iOS untested, root cause applicability to Android 14+) and says how to resolve them (separate rollouts, engineering confirmation).

Checks every criterion for every platformRightWrongRight

Haiku 5.5 · API

Table correctly computes each criterion per platform against thresholds; identifies Android 12 and older as failing all four, and Android 13 as failing the sync-error criterion.

Gemini 3.5 Flash-Lite · Gemini

Android 13's error-free sync is 98.9%, below the 99% threshold, yet the table marks it as 'GO WITH CONDITIONS' instead of 'NO-GO' for that criterion, misapplying the hard launch criteria.

Opus 5.5 · Claude

The table correctly computes each criterion for every platform against control and thresholds, and correctly identifies Android 12 and older as failing (and Android 13 as failing the sync bar, which is accurate per the data).

Limits the downside of being wrongRightWrongRight

Haiku 5.5 · API

Specifies daily monitoring with concrete rollback thresholds, flushing of offline edits before disabling, and segmentation of the email to limit irreversible comms.

Gemini 3.5 Flash-Lite · Gemini

Does not name a post-launch signal, a threshold that would pause or roll back, or flag irreversible actions; only mentions 'expedited telemetry monitoring' without specifics.

Opus 5.5 · Claude

Specifies post-launch monitoring of crash-free sessions, conflict tickets and sync errors, a rollback trigger if any segment crosses a threshold, and flags the irreversible email communication.

Judges the launch against a baseline set nowRightWrongRight

Haiku 5.5 · API

Uses control metrics as baselines and sets daily alert thresholds with rollback action, effectively defining success metrics and review points.

Gemini 3.5 Flash-Lite · Gemini

Does not name a success metric with its current baseline from the data, nor a review point with what each result would lead to.

Opus 5.5 · Claude

Names the metrics (crash-free sessions, conflict tickets, sync errors), implies the baseline from the agreed launch criteria, and sets a daily monitoring period with a rollback trigger.

All got right 5

Addresses the actual decisionRightRightRight

Haiku 5.5 · API

Commits unambiguously to a partial launch (iOS 17+, Android 14+), holds Android 13, and turns off Android 12 and older, with rollback triggers that would change the call.

Gemini 3.5 Flash-Lite · Gemini

Commits early to a segmented go/no-go call per platform, framed for the leadership team, and states that a refactored storage handler would change the no-go for Android 12 and older.

Opus 5.5 · Claude

Commits early to a clear, conditional go for iOS 17+ and Android 14+, hold for Android 13, and no-go for Android 12 and older, and states what would change the call (feature flag targeting capability, engineering sign-off).

Avoids unsupported claimsRightRightRight

Haiku 5.5 · API

Uses wording like 'suggests' for the storage API inference, and the data-loss judgment is grounded in the supplied criterion; no confident unsupported factual claims.

Gemini 3.5 Flash-Lite · Gemini

Interpretive statements like 'iOS is Fully Vetted' are clearly opinion, and causal claims (legacy storage API) are directly supported by the incident log.

Opus 5.5 · Claude

Interpretations like 'expected when people can now edit offline' are not presented as established fact, and no unsupported factual claims are made.

Produces the required deliverableRightRightRight

Haiku 5.5 · API

Provides a complete, usable recommendation with table, go/no-go calls, monitoring, and email handling, appropriate for a launch review.

Gemini 3.5 Flash-Lite · Gemini

The memo is complete, within word limit, contains the required table, and a leadership team could act on the segmented recommendation with light edits.

Opus 5.5 · Claude

The recommendation is complete, in the right form for the PM, within the word limit, and could be acted on with light edits.

Treats the overwrites as data lossRightRightRight

Haiku 5.5 · API

Explicitly treats the recovered overwritten edits as a breach of the no-data-loss criterion and makes it a decisive condition for Android 12 and older.

Gemini 3.5 Flash-Lite · Gemini

Explicitly treats the two overwritten notes on Android 11 as a breach of the no-data-loss criterion and makes fixing the storage API a condition for that platform.

Opus 5.5 · Claude

Treats the overwritten notes on Android 11 as a breach of the no-data-loss criterion and makes fixing the storage-API issue a condition for that platform.

Gets the base of every number rightRightRightRight

Haiku 5.5 · API

All percentages, differences, and per‑1,000 rates are computed correctly from the rollout data and control values.

Gemini 3.5 Flash-Lite · Gemini

All percentage differences are computed correctly from the control and offline-sync values in the CSV, and the base (percentage points) is clear.

Opus 5.5 · Claude

All derived figures (gaps, rates) are computed correctly from the supplied data, and bases (percentage points, per 1,000 users) are clear.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 63% of 68 checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	Sonnet 5.5withAPI	87.8	100.0	2	None
2	Haiku 5.5withAPI	82.5	100.0	2	None
3	GPT-6.1 SolwithAPI	86.0	95.8	2	None
4	Opus 5.5withClaude	84.3	96.7	2	None
5	GPT-6 AstrawithChatGPT	83.8	92.5	2	None
6	GPT-6 LunawithAPI	74.8	80.8	2	None
7	Gemini 3.8 FlashwithAPI	64.9	67.5	2	1 capped
8	Gemini 3.5 Flash-LitewithGemini	48.2	59.2	2	None

What you get for the cost

Haiku 5.5 · API scores within 2 points of Sonnet 5.5 · API at about a tenth of the cost.

Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)At least one critical failure

Task score and relative cost by setup
Setup	Task score	Relative cost (cheapest = 1)
GPT-6 Luna · API	77	1.0×
Gemini 3.5 Flash-Lite · Gemini	52	2.7× (partly estimated)
Haiku 5.5 · API	92	2.8×
Gemini 3.8 Flash · API	55	11×
GPT-6.1 Sol · API	91	12×
Sonnet 5.5 · API	94	37×
GPT-6 Astra · ChatGPT	89	46× (partly estimated)
Opus 5.5 · Claude	91	59× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Deciding whether a feature ships on the planned date, and on what conditions.

Why it matters

Launch meetings reward optimism. A good call checks each agreed criterion, weighs a real risk against a real gain, and says exactly what would change the answer. A weak one rubber-stamps the launch or blocks it on noise.

What good looks like

Checks each agreed criterion against the evidence
Makes one clear call, with conditions if needed
Separates risks that block launch from risks that can be managed
Says what would change the call
Says how to limit the damage if the call is wrong

Deliberately not measured

Rollout engineering detail
Project-plan formatting

Capability tested

Deciding against agreed launch criteria

The failure we’re looking for

Rubber-stamps a launch that misses an agreed bar, or blocks it on noise

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

A launch memo from supplied evidence · Staff level: a regional call from an attached data workbook

What AI gets right here, and what you’ll still have to catch

Reliably right

Where it slips

How it’s graded

Limits the downside of being wrong

Judges the launch against a baseline set now

Plus our standard checks

The tasks

The outputs

Output

Recommendation: offline sync to iOS 17+ and Android 14+ only

Why not a global launch

Before Friday

Monday's email

Summary

Claims the judge couldn’t find in the brief

Grades and run details

Decision model checks

Run

Output

Executive Summary

Launch Criteria Status Matrix

Key Findings & Rationale

Action Plan for Monday’s Launch

Grades and run details

Decision model checks

Run

Output

Offline sync launch recommendation

Criteria check (25% rollout, 14 days)

Why

Conditions for Friday's go

Marketing email

Next steps

Grades and run details

Decision model checks

Run

Check by check

Where they differ 6

All got right 5

Results

What you get for the cost

About the task

The PM job

Why it matters

What good looks like

Deliberately not measured