Tasks / Product Craft

Make the launch call

Can the model make a clear go / no-go call from mixed launch evidence, checked against the criteria agreed up front?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 67% were usable with at most a quick edit.

Reliably right

Produces the required deliverable100% pass
Provides a complete recommendation with table, reasoning, email plan, and next steps, usable by a product manager with light edits.
Sonnet 5.5 · API · Offline, on older phones
Finds the Scotland breach100% pass
Finds Scotland's refund breach, ties it to the supplier change, notes the acceptance dip, and holds Scotland.
GPT-6.1 Sol · API · Go/no-go for AI substitutions, from the rollout data
Treats the overwrites as data loss100% pass
Treats the Android 11 overwrites as a breach of the no-data-loss criterion and makes a fix a condition for that platform.
Sonnet 5.5 · API · Offline, on older phones

Where it slips

Judges the launch against a baseline set now51% pass
Does not name a specific review point or when the team will review results to decide next steps; only gives a pause trigger without a scheduled evaluation.
GPT-6.1 Sol · API · Offline, on older phones
Checks every criterion for every platform64% pass
Android 13's error-free sync is 98.9%, below the 99% threshold, yet the table marks it as 'GO WITH CONDITIONS' instead of 'NO-GO' for that criterion, misapplying the hard launch criteria.
Gemini 3.5 Flash-Lite · Gemini · Offline, on older phones
Gets the base of every number right67% pass
The 80.5% user base proportion is miscalculated; correct proportion from the data is 79.5%.
Gemini 4 Argon · API · Offline, on older phones

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

Limits the downside of being wrong
Does the call say how exposure is limited if it's wrong: what is watched after launch, the threshold that would pause or roll back, and which parts can't be undone?
Passes when Names the post-launch signal, a threshold or trigger, and the action, and flags anything irreversible (customer communications, pricing, migrations).
Karri SaarinenLenny’s Podcast
Judges the launch against a baseline set now
Does the call name the metric and the current baseline the launch will be judged against, and when the team will review the result to decide what's next?
Passes when Names the success metric, its baseline from the material, and a review point with what each result would lead to.
Todd JacksonLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for Lumen's mobile apps. The launch review for offline sync is on Friday, and marketing has an email booked for Monday to every user. Write your recommendation, in no more than 700 words, with a table checking each launch criterion for each platform: go, no-go or go with conditions, and why. The rollout data, the criteria and the incident log are attached.

What the model was given5 items: About Lumen, launch_criteria.md, rollout_25pct_14_days.csv, incidents.md, the_decision.md

About LumenA notes app for teams, on iOS, Android and the web. Offline sync lets people edit notes without a connection and merges changes when they reconnect.

launch_criteria.md6 lines · Download

# Launch criteria (agreed before the rollout)

1. Crash-free sessions no more than 0.2 points below control, per platform.
2. Sync-conflict tickets under 2 per 1,000 users.
3. Notes synced without error at least 99%.
4. No data loss incidents.

rollout_25pct_14_days.csv8 rows · Download

platform,arm,users,crash_free_sessions_pct,sync_conflict_tickets_per_1000_users,notes_synced_without_error_pct
iOS 17+,control,123600,99.62,0.3,99.5
iOS 17+,offline sync,41200,99.58,0.6,99.4
Android 14+,control,68400,99.55,0.3,99.5
Android 14+,offline sync,22800,99.49,0.8,99.2
Android 13,control,27300,99.4,0.3,99.5
Android 13,offline sync,9100,99.31,1.1,98.9
…

incidents.md4 lines · Download

# Incidents during the 25% rollout (14 days)

- 2 reports of a note's edits being overwritten after a conflict, both on Android 11; engineering recovered both from backups. Root cause: an older Android storage API.
- 1 report of slow sync on iOS, not reproduced.

the_decision.md1 lines · Download

The launch review is Friday: roll offline sync out to 100%, or not. Marketing has a launch email booked for Monday to every user.

What a strong answer doesThe answer key the graders mark against

Checks each criterion per platform. iOS 17+, Android 14+ and Android 13 pass: crash-free sessions within 0.2 points of control (iOS -0.04, Android 14+ -0.06, Android 13 -0.09), sync-conflict tickets under 2 per 1,000, notes synced without error at least 99%. Android 12 and older fails three criteria: crash-free down 0.67 points against control, 4.9 conflict tickets per 1,000, 96.1% synced without error; and the two overwritten-note incidents were on Android 11, which breaks 'no data loss' (both recovered from backup, root cause an older storage API). Recommends go for iOS, Android 13 and newer, and no-go for Android 12 and older until the storage issue is fixed and re-tested; the Monday email should only go to users on supported versions, or wait. States what's monitored after launch and the threshold that would pause it.

Critical failures (cap the score at 40)

Recommends rolling out to 100% on every platform

Task

Staff-level · v1.1

The outputs

9 setups answered this brief.

Sonnet 5.5API98Usable with a quick edit Opus 5.5Claude94Usable with a quick edit Haiku 5.5API94Usable with a quick edit GPT-6.1 SolAPI92Usable with a quick edit GPT-6 AstraChatGPT92Usable with a quick edit GPT-6 LunaAPI75Needs more work Gemini 4 ArgonAPI71Usable with a quick edit Gemini 3.8 FlashAPI69Needs more work Gemini 3.5 Flash-LiteGemini58Needs more work

Gemini 4 ArgonAPI

Usable with a quick edit7 of 11 checks passed

The platform decisions and table are correct; a PM could quickly fix the 80.5% arithmetic and add a monitoring plan without redoing the analysis.

Output

To: Launch Review Committee From: Product Manager, Mobile Apps Date: Friday (Launch Review) Subject: Offline Sync 100% Rollout Recommendation

Executive Recommendation: Go with Conditions (Segmented Rollout)

Based on our 14-day, 25% rollout data, I recommend a Go for iOS 17+ and Android 14+, which represent 80.5% of our mobile user base.

I recommend a No-Go for Android 12 and older due to severe stability and data loss issues, and a Go with Conditions (or short delay) for Android 13, which narrowly missed our sync error threshold by 0.1%.

Consequently, Marketing must not send the Monday launch email to every user. We must either segment the email to eligible OS versions or adjust the messaging to reflect system requirements (iOS 17+ / Android 14+).

---

Launch Criteria Evaluation by Platform

Platform (% of Total Users)	1. Crash-Free Sessions (Max -0.2 pts vs. Control)	2. Conflict Tickets (< 2.0 per 1k users)	3. Sync Success Rate (≥ 99.0%)	4. Data Loss Incidents (0 incidents)	Decision & Rationale
:---	:---	:---	:---	:---	:---
iOS 17+<br>(51.2%)	PASS<br>99.58% vs 99.62%<br>(-0.04 pts)	PASS<br>0.6	PASS<br>99.4%	PASS<br>0 incidents (1 unrep. slow sync)	GO<br>Meets all criteria comfortably.
Android 14+<br>(29.3%)	PASS<br>99.49% vs 99.55%<br>(-0.06 pts)	PASS<br>0.8	PASS<br>99.2%	PASS<br>0 incidents	GO<br>Meets all criteria comfortably.
Android 13<br>(11.7%)	PASS<br>99.31% vs 99.40%<br>(-0.09 pts)	PASS<br>1.1	FAIL<br>98.9%<br>(-0.1% below target)	PASS<br>0 incidents	GO WITH CONDITIONS<br>Borderline miss on sync errors (98.9%). Safe on data loss and crashes.
Android 12 & older<br>(9.5%)	FAIL<br>98.71% vs 99.38%<br>(-0.67 pts)	FAIL<br>4.9	FAIL<br>96.1%	FAIL<br>2 overwritten notes on Android 11	NO-GO<br>Fails every criterion. Legacy storage API causes data overwrites.

---

Detailed Rationale & Next Steps

1. Greenlight iOS 17+ and Android 14+ (Roll out to 100% on Monday) Both modern platforms passed every gate with healthy margins. The single report of slow sync on iOS could not be reproduced and did not impact overall sync success rates (99.4%). These cohorts are ready for a 100% rollout.

2. Block & Roll Back Android 12 and Older (Immediate Action) Android 12 and older failed all four launch criteria. Most critically, an older Android storage API caused two incidents on Android 11 where conflict resolution overwrote user edits (requiring manual backup recovery by engineering). For a collaborative notes app, data loss is a dealbreaker. * Action: Immediately roll back the 25% test cohort on Android ≤12 to 0%. Engineering will scope whether fixing the legacy storage API is worth the investment for 9.5% of our base, or if we officially deprecate offline sync for Android ≤12.

3. Android 13 Conditions Android 13 passed stability, conflict ticket, and zero-data-loss criteria, but hit 98.9% on error-free syncs (0.1 points short of our 99.0% gate). * Condition for Go: If Engineering confirms before EOD today that Android 13 does not use the legacy storage API that caused the Android 11 data loss, and that the 1.1% error rate fails gracefully (i.e., retries on next connection without corrupting data), we can include Android 13 in Monday’s launch while fast-following a patch. * Alternative: If there is any architectural overlap with the Android 11 bug, hold Android 13 at 25% until patched.

4. Marketing & Monday Email Plan Because we are holding back at least 9.5% (and up to 21.2%) of users on older Android versions, we cannot send a blanket announcement to all users without generating support tickets and frustration. Marketing has two options for Monday: * Option A (Recommended): Segment the email list to send only to users on iOS 17+ and Android 14+ (and Android 13, if cleared today), plus Web users. * Option B: Send to all users, but explicitly state in the email body and help docs: "Offline sync is rolling out today for iOS 17+ and Android 14+."

Claims the judge couldn’t find in the brief

iOS 17+ and Android 14+ represent 80.5% of our mobile user base.

Grades and run details

Decision model 73 · LLM judge 67

Decision model checks

passUses the supplied evidence correctly23%
passAddresses the actual decision62%
passRespects explicit constraints43%
passIdentifies material uncertainty97%
partialAvoids unsupported claims25%
passProduces the required deliverable61%
partialChecks every criterion for every platform52%
passTreats the overwrites as data loss100%
partialLimits the downside of being wrong70%
failGets the base of every number right13%
partialJudges the launch against a baseline set now30%

Run

Run: #1
API response time: 12 s
Submitted: 13 Oct 2026

Sonnet 5.5API

Usable with a quick edit11 of 11 checks passed

The output is factually correct, well-structured, and includes all required elements; a PM could act on it with at most minor wording tweaks.

Output

Recommendation: partial go

Ship offline sync to 100% on iOS 17+ and Android 14+. Do not ship it on Android 13 or Android 12 and older. Segment Monday's email so it doesn't promise the feature to users who won't have it.

Criteria check (25% rollout, 14 days)

Platform	1. Crash-free (≥ control −0.2)	2. Conflict tickets (<2/1,000)	3. Synced w/o error (≥99%)	4. No data loss	Verdict
iOS 17+	Go: −0.04	Go: 0.6	Go: 99.4%	Go: none (one slow-sync report, not reproduced)	Go
Android 14+	Go: −0.06	Go: 0.8	Go: 99.2%	Go: none	Go
Android 13	Go: −0.09	Go: 1.1, but over half the limit	No-go: 98.9%, misses by 0.1	Go: none reported	No-go for now
Android 12 and older	No-go: −0.67	No-go: 4.9, about 2.5× the limit	No-go: 96.1%	No-go: 2 overwrites on Android 11	No-go

Reasoning

iOS 17+ and Android 14+ pass every criterion with margin. Together they are about 80% of users. The iOS slow-sync report wasn't reproduced, so I'd watch it during the ramp but not block on it.

Android 13 misses one criterion by 0.1 points. That is small, but we agreed the criteria before seeing the data, and I don't want to bend them the first time a result is inconvenient. The trend also worries me. Sync success falls with each older Android tier (99.2 → 98.9 → 96.1), and conflict tickets rise (0.8 → 1.1 → 4.9). This looks like a gradient tied to the OS, not noise, so Android 13 sits on the slope. I'd hold it at its current exposure, find out why it's degrading, and re-measure. If it reaches 99% or better with conflicts still under 2, it can go.

Android 12 and older fails all four criteria, and it is not close. Crash-free sessions are 0.67 points below control, more than three times the allowed gap. Conflict tickets are about 2.5× the limit, and sync success is 3 points under the bar. The two data-loss incidents were edits overwritten after a conflict, on Android 11. Engineering recovered both from backups, but that doesn't make them acceptable. We only recovered them because users reported them, and silent overwrites may not have been reported. The root cause is an older Android storage API, so this is a structural problem and not a quick patch. I'd also disable offline sync for the users currently exposed on Android 11 and below until it's fixed. Data loss in a notes app is the failure users won't forgive.

The Monday email

An email to every user would promise a feature to roughly 20% of users (Android 13 and older, about 66,000 of 322,000) who won't have it. That means confusion, support tickets, and a bad first impression of the feature. My recommendation is:

Send Monday to iOS 17+ and Android 14+ users only, and only if the ramp over the weekend stays within the criteria. Marketing should confirm it can segment by platform and OS version.
Hold the email for Android 13 and older. Send it when each segment passes, or send a "coming to your device soon" version if marketing wants one.
Check the ramp Monday morning. If crash-free sessions or sync success slip on either approved platform, pause the email and the ramp.

Conditions and next steps

Android 12 and older: Engineering should scope a fix for the storage API issue, and we should re-test in a new, smaller cohort. I'd also like the data split into Android 12 versus 11 and below. The incidents were all on Android 11, so Android 12 itself may be fine, but this bucket can't tell us.
Android 13: Investigate the sync errors and re-measure in 1–2 weeks.
Data-loss monitoring: Add a check for overwrite after conflict, so we aren't relying on users to tell us.

Caveats

The data covers 14 days at 25%. The Android 13 miss is small enough that more data could move it either way.
Per-platform samples are decent (7,400 to 41,200 users), but two data-loss incidents is a tiny count. Treat it as a signal and not an estimate of the true rate.

Grades and run details

Decision model 95 · LLM judge 100

Decision model checks

passUses the supplied evidence correctly49%
passAddresses the actual decision83%
passRespects explicit constraints48%
passIdentifies material uncertainty100%
passAvoids unsupported claims34%
passProduces the required deliverable85%
partialChecks every criterion for every platform48%
passTreats the overwrites as data loss100%
passLimits the downside of being wrong94%
passGets the base of every number right38%
passJudges the launch against a baseline set now37%

Run

Run: #1
API response time: 28 s
Submitted: 6 Oct 2026

Opus 5.5Claude

Usable with a quick edit10 of 11 checks passed

The output is clear, data-backed, and includes all required elements; a PM could use it with minimal edits, without rechecking evidence or redoing reasoning.

Output

Offline sync launch recommendation

Recommendation: Go with conditions on Friday, for iOS 17+ and Android 14+ only. Hold Android 13. Turn the feature off for Android 12 and older. Do not send Monday's email to every user.

Two segments meet every criterion: iOS 17+ and Android 14+. Android 13 fails the sync-reliability bar. Android 12 and older fails three of four criteria and is where both data-loss incidents occurred. The criteria were agreed before the rollout, so I don't recommend relaxing them now that we have results.

Criteria check (25% rollout, 14 days)

Platform	1. Crash-free gap vs control (max 0.2 pts)	2. Conflict tickets (under 2 per 1,000)	3. Notes synced without error (≥99%)	4. No data loss	Decision
iOS 17+	✅ 0.04 pts (99.58 vs 99.62)	✅ 0.6	✅ 99.4%	✅ None. One slow-sync report, not reproduced	Go
Android 14+	✅ 0.06 pts (99.49 vs 99.55)	✅ 0.8	✅ 99.2%	✅ None reported	Go with conditions: engineering confirms the Android 11 storage-API bug can't occur on 14+
Android 13	✅ 0.09 pts (99.31 vs 99.40)	✅ 1.1	❌ 98.9%	✅ None reported	No-go: misses the agreed sync bar. Hold at 25% and fix
Android 12 and older	❌ 0.67 pts (98.71 vs 99.38)	❌ 4.9, about 16× control	❌ 96.1%	❌ 2 incidents (Android 11), recovered only from backups	No-go: turn off for current rollout users
Web	No data	No data	No data	No data	Not assessable: wasn't in the rollout

Why

iOS 17+ and Android 14+ are clean. Every metric clears its threshold with margin. The crash gaps are a fraction of the 0.2-point allowance. Conflict tickets are about 2–3× control, which is expected when people can now edit offline, and still well under the limit. The only condition on Android 14+ is confirmation that the overwrite bug is tied to the older storage API and can't occur there.

Android 13 is a near miss, but still a miss. It passes three criteria. Its 98.9% sync rate is just under the 99% bar. It is also trending worse than 14+ on every metric. Shipping it to 100% would mean overriding a criterion we agreed in advance. I'd rather fix the cause and re-review it in a follow-up launch.

Android 12 and older is unsafe. Roughly 1 in 25 notes fails to sync. The crash gap is more than three times the allowance. Two users had edits overwritten. We recovered those edits only because engineering pulled them from backups, which doesn't scale, and a user who never reports the problem simply loses their work. I recommend turning offline sync off now for the 7,400 users on this segment, not just excluding them from 100%.

Gaps in coverage: - Web has no rollout data at all. - iOS below 17 wasn't in the rollout, so it is untested.

Both should stay off until they have their own rollout.

Conditions for Friday's go

OS-version targeting. The feature flag can target by platform and OS version, so 100% means 100% of iOS 17+ and Android 14+ only. If it can't, the answer for Friday is no-go.
Engineering sign-off on Android 14+. Written confirmation that the overwrite root cause doesn't apply.
Off for Android 12 and older now. Done before or at launch, with a check that no user loses offline edits made while it was on.
Monitoring after launch. We watch crash-free sessions, conflict tickets and sync errors daily for two weeks. We roll back if any segment crosses a threshold.

Marketing email

An email to every user on Monday would promote offline sync to people who can't get it, including all web users, older iOS and Android 13 and below. For the oldest Android group, it would point them at the version we know loses data. My options, in order of preference:

Segment the email to iOS 17+ and Android 14+ users, if marketing can target by OS version.
Postpone the email until we can segment it, or until more platforms qualify.
Change the copy to say "now on iOS and newer Android, more platforms coming." This is a last resort, because it still lands with users who can't use the feature.

Next steps

Android 13: engineering investigates the sync-error rate, then we re-review.
Android 12 and older: fix the storage-API issue and run a fresh rollout before it's considered again.
Web and iOS below 17: start their own 25% rollouts against the same criteria.

Grades and run details

Decision model 86 · LLM judge 100

Decision model checks

passUses the supplied evidence correctly46%
passAddresses the actual decision85%
passRespects explicit constraints53%
passIdentifies material uncertainty100%
partialAvoids unsupported claims35%
passProduces the required deliverable86%
failChecks every criterion for every platform25%
passTreats the overwrites as data loss100%
passLimits the downside of being wrong95%
passGets the base of every number right67%
passJudges the launch against a baseline set now35%

Run

Run: #1
Time to output: 30 s
Submitted: 5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 5

Uses the supplied evidence correctlyMixedRightRight

Gemini 4 Argon · API

The claim that iOS 17+ and Android 14+ represent 80.5% of mobile users is incorrect; the correct proportion from the supplied data is 79.5%.

Sonnet 5.5 · API

All factual claims about the current situation are directly supported by the supplied data or simple arithmetic from it.

Opus 5.5 · Claude

All factual claims about the current situation are taken directly from the supplied rollout data and incident log, with no invented facts.

Checks every criterion for every platformRightRightMixed

Gemini 4 Argon · API

Table correctly computes each criterion per platform, identifying Android 12 and older as the only failing platform across all criteria.

Sonnet 5.5 · API

Table correctly computes each criterion for every platform against control and thresholds, and correctly identifies Android 12 and older as failing.

Opus 5.5 · Claude

The table correctly computes each criterion for every platform against control and thresholds, and correctly identifies Android 12 and older as failing (and Android 13 as failing the sync bar, which is accurate per the data).

Limits the downside of being wrongWrongRightRight

Gemini 4 Argon · API

Does not name a post-launch monitoring signal, threshold, or action for the greenlit platforms, nor flag irreversible customer communications beyond email segmentation.

Sonnet 5.5 · API

Specifies post-launch monitoring (crash-free, sync success), a trigger (slip), and action (pause email and ramp), and flags the irreversible email communication.

Opus 5.5 · Claude

Specifies post-launch monitoring of crash-free sessions, conflict tickets and sync errors, a rollback trigger if any segment crosses a threshold, and flags the irreversible email communication.

Gets the base of every number rightWrongRightRight

Gemini 4 Argon · API

The 80.5% user base proportion is miscalculated; correct proportion from the data is 79.5%.

Sonnet 5.5 · API

All derived figures (differences, percentages, user shares) are computed correctly from the supplied data, with no ambiguous bases.

Opus 5.5 · Claude

All derived figures (gaps, rates) are computed correctly from the supplied data, and bases (percentage points, per 1,000 users) are clear.

Judges the launch against a baseline set nowWrongRightRight

Gemini 4 Argon · API

Does not name a post-launch review timeline or baseline metric beyond the existing launch criteria, nor what each result would lead to.

Sonnet 5.5 · API

Names the success metrics (crash-free sessions, sync success), implies the baseline from the go criteria, and sets a review point (Monday morning) with a clear action if they slip.

Opus 5.5 · Claude

Names the metrics (crash-free sessions, conflict tickets, sync errors), implies the baseline from the agreed launch criteria, and sets a daily monitoring period with a rollback trigger.

All got right 6

Addresses the actual decisionRightRightRight

Gemini 4 Argon · API

Commits to a clear per-platform recommendation (Go/No-Go/Go with Conditions) early, and states what would change for Android 13.

Sonnet 5.5 · API

Commits early to a clear partial-go recommendation (ship to iOS 17+ and Android 14+, hold Android 13 and older) and states what would change each call.

Opus 5.5 · Claude

Commits early to a clear, conditional go for iOS 17+ and Android 14+, hold for Android 13, and no-go for Android 12 and older, and states what would change the call (feature flag targeting capability, engineering sign-off).

Respects explicit constraintsRightRightRight

Gemini 4 Argon · API

The recommendation is within 700 words, contains the required table, and addresses a launch review committee.

Sonnet 5.5 · API

Output is under 700 words, includes the required table with go/no-go per platform, and addresses the launch review decision and Monday email.

Opus 5.5 · Claude

The output is under 700 words, includes the required table with go/no-go/go with conditions per platform, and is written as a recommendation for the PM.

Identifies material uncertaintyRightRightRight

Gemini 4 Argon · API

Identifies key uncertainty about Android 13's storage API overlap and defines the condition for inclusion.

Sonnet 5.5 · API

Names the small Android 13 miss, the need to split Android 12 vs 11, and the tiny data-loss count as uncertainties, and says how to resolve them.

Opus 5.5 · Claude

Identifies gaps (web and older iOS untested, root cause applicability to Android 14+) and says how to resolve them (separate rollouts, engineering confirmation).

Avoids unsupported claimsRightRightRight

Gemini 4 Argon · API

No interpretations or causes are presented as established fact beyond what's supported; the root cause is from the incident log.

Sonnet 5.5 · API

Interpretations like the OS gradient are presented as hypotheses ('looks like'), and no unsupported facts are stated as certain.

Opus 5.5 · Claude

Interpretations like 'expected when people can now edit offline' are not presented as established fact, and no unsupported factual claims are made.

Produces the required deliverableRightRightRight

Gemini 4 Argon · API

The memo includes the table, clear calls per platform, and actionable steps for marketing, making it immediately usable.

Sonnet 5.5 · API

Provides a complete recommendation with table, reasoning, email plan, and next steps, usable by a product manager with light edits.

Opus 5.5 · Claude

The recommendation is complete, in the right form for the PM, within the word limit, and could be acted on with light edits.

Treats the overwrites as data lossRightRightRight

Gemini 4 Argon · API

Treats the overwritten notes as a data-loss incident and uses it as a reason for no-go for Android 12 and older.

Sonnet 5.5 · API

Treats the Android 11 overwrites as a breach of the no-data-loss criterion and makes a fix a condition for that platform.

Opus 5.5 · Claude

Treats the overwritten notes on Android 11 as a breach of the no-data-loss criterion and makes fixing the storage-API issue a condition for that platform.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 62% of 68 checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	Sonnet 5.5withAPI	92.4	100.0	2	None
2	GPT-6.1 SolwithAPI	91.9	95.8	2	None
3	Haiku 5.5withAPI	84.3	100.0	2	None
4	GPT-6 AstrawithChatGPT	91.9	92.5	2	None
5	Opus 5.5withClaude	84.3	96.7	2	None
6	GPT-6 LunawithAPI	80.2	80.8	2	None
7	Gemini 4 ArgonwithAPI	70.3	73.3	2	None
8	Gemini 3.5 Flash-LitewithGemini	55.0	59.2	2	None
9	Gemini 3.8 FlashwithAPI	64.4	67.5	2	1 capped

What you get for the cost

Haiku 5.5 · API scores within 3 points of Sonnet 5.5 · API at about a tenth of the cost.

Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)At least one critical failure

Task score and relative cost by setup
Setup	Task score	Relative cost (cheapest = 1)
GPT-6 Luna · API	79	1.0×
Gemini 3.5 Flash-Lite · Gemini	55	2.7× (partly estimated)
Haiku 5.5 · API	93	2.8×
Gemini 3.8 Flash · API	54	11×
GPT-6.1 Sol · API	94	12×
Gemini 4 Argon · API	71	14×
Sonnet 5.5 · API	96	37×
GPT-6 Astra · ChatGPT	93	46× (partly estimated)
Opus 5.5 · Claude	91	59× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Deciding whether a feature ships on the planned date, and on what conditions.

Why it matters

Launch meetings reward optimism. A good call checks each agreed criterion, weighs a real risk against a real gain, and says exactly what would change the answer. A weak one rubber-stamps the launch or blocks it on noise.

What good looks like

Checks each agreed criterion against the evidence
Makes one clear call, with conditions if needed
Separates risks that block launch from risks that can be managed
Says what would change the call
Says how to limit the damage if the call is wrong

Deliberately not measured

Rollout engineering detail
Project-plan formatting

Capability tested

Deciding against agreed launch criteria

The failure we’re looking for

Rubber-stamps a launch that misses an agreed bar, or blocks it on noise

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

A launch memo from supplied evidence · Staff level: a regional call from an attached data workbook

What AI gets right here, and what you’ll still have to catch

Reliably right

Where it slips

How it’s graded

Limits the downside of being wrong

Judges the launch against a baseline set now

Plus our standard checks

The tasks

The outputs

Output

Executive Recommendation: Go with Conditions (Segmented Rollout)

Launch Criteria Evaluation by Platform

Detailed Rationale & Next Steps

Claims the judge couldn’t find in the brief

Grades and run details

Decision model checks

Run

Output

Recommendation: partial go

Criteria check (25% rollout, 14 days)

Reasoning

The Monday email

Conditions and next steps

Caveats

Grades and run details

Decision model checks

Run

Output

Offline sync launch recommendation

Criteria check (25% rollout, 14 days)

Why

Conditions for Friday's go

Marketing email

Next steps

Grades and run details

Decision model checks

Run

Check by check

Where they differ 5

All got right 6

Results

What you get for the cost

About the task

The PM job

Why it matters

What good looks like

Deliberately not measured