Tasks / Product Craft

Make the launch call

Can the model make a clear go / no-go call from mixed launch evidence, checked against the criteria agreed up front?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 69% were usable with at most a quick edit.

Reliably right

Finds the Scotland breach100% pass
Finds Scotland's refund breach (+66% overall, +143% since 15 Oct), ties it to the supplier change, notes acceptance dip, and holds Scotland.
Sonnet 5.5 · API · Go/no-go for AI substitutions, from the rollout data
Treats the overwrites as data loss100% pass
Treats the overwritten notes on Android 11 as a breach of the no-data-loss criterion and makes fixing the storage-API issue a condition for that platform.
Opus 5.5 · Claude · Offline, on older phones
Addresses the actual decision98% pass
Commits to a clear no-go for 2 Nov, conditional go from 4 Nov for four regions, and no-go for Scotland, with conditions that would change it.
Sonnet 5.5 · API · Go/no-go for AI substitutions, from the rollout data

Where it slips

Limits the downside of being wrong63% pass
It does not name a specific post-launch signal, a threshold that would trigger pause or rollback, or flag any irreversible actions.
GPT-6 Luna · API · Go/no-go for AI substitutions, from the rollout data
Judges the launch against a baseline set now66% pass
Does not name a specific review point or when the team will review results to decide next steps; only gives a pause trigger without a scheduled evaluation.
GPT-6.1 Sol · API · Offline, on older phones
Checks every criterion for every platform69% pass
Android 13's error-free sync is 98.9%, below the 99% threshold, yet the table marks it as 'GO WITH CONDITIONS' instead of 'NO-GO' for that criterion, misapplying the hard launch criteria.
Gemini 3.5 Flash-Lite · Gemini · Offline, on older phones

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

Limits the downside of being wrong
Does the call say how exposure is limited if it's wrong: what is watched after launch, the threshold that would pause or roll back, and which parts can't be undone?
Passes when Names the post-launch signal, a threshold or trigger, and the action, and flags anything irreversible (customer communications, pricing, migrations).
Karri SaarinenLenny’s Podcast
Judges the launch against a baseline set now
Does the call name the metric and the current baseline the launch will be judged against, and when the team will review the result to decide what's next?
Passes when Names the success metric, its baseline from the material, and a review point with what each result would lead to.
Todd JacksonLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for Lumen's mobile apps. The launch review for offline sync is on Friday, and marketing has an email booked for Monday to every user. Write your recommendation, in no more than 700 words, with a table checking each launch criterion for each platform: go, no-go or go with conditions, and why. The rollout data, the criteria and the incident log are attached.

What the model was given5 items: About Lumen, launch_criteria.md, rollout_25pct_14_days.csv, incidents.md, the_decision.md

About LumenA notes app for teams, on iOS, Android and the web. Offline sync lets people edit notes without a connection and merges changes when they reconnect.

launch_criteria.md6 lines · Download

# Launch criteria (agreed before the rollout)

1. Crash-free sessions no more than 0.2 points below control, per platform.
2. Sync-conflict tickets under 2 per 1,000 users.
3. Notes synced without error at least 99%.
4. No data loss incidents.

rollout_25pct_14_days.csv8 rows · Download

platform,arm,users,crash_free_sessions_pct,sync_conflict_tickets_per_1000_users,notes_synced_without_error_pct
iOS 17+,control,123600,99.62,0.3,99.5
iOS 17+,offline sync,41200,99.58,0.6,99.4
Android 14+,control,68400,99.55,0.3,99.5
Android 14+,offline sync,22800,99.49,0.8,99.2
Android 13,control,27300,99.4,0.3,99.5
Android 13,offline sync,9100,99.31,1.1,98.9
…

incidents.md4 lines · Download

# Incidents during the 25% rollout (14 days)

- 2 reports of a note's edits being overwritten after a conflict, both on Android 11; engineering recovered both from backups. Root cause: an older Android storage API.
- 1 report of slow sync on iOS, not reproduced.

the_decision.md1 lines · Download

The launch review is Friday: roll offline sync out to 100%, or not. Marketing has a launch email booked for Monday to every user.

What a strong answer doesThe answer key the graders mark against

Checks each criterion per platform. iOS 17+, Android 14+ and Android 13 pass: crash-free sessions within 0.2 points of control (iOS -0.04, Android 14+ -0.06, Android 13 -0.09), sync-conflict tickets under 2 per 1,000, notes synced without error at least 99%. Android 12 and older fails three criteria: crash-free down 0.67 points against control, 4.9 conflict tickets per 1,000, 96.1% synced without error; and the two overwritten-note incidents were on Android 11, which breaks 'no data loss' (both recovered from backup, root cause an older storage API). Recommends go for iOS, Android 13 and newer, and no-go for Android 12 and older until the storage issue is fixed and re-tested; the Monday email should only go to users on supported versions, or wait. States what's monitored after launch and the threshold that would pause it.

Critical failures (cap the score at 40)

Recommends rolling out to 100% on every platform

Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Opus 5.5Claude94Usable with a quick edit Sonnet 5.5API94Usable with a quick edit Haiku 5.5API94Usable with a quick edit GPT-6.1 SolAPI90Usable with a quick edit GPT-6 AstraChatGPT88Usable with a quick edit GPT-6 LunaAPI75Needs more work Gemini 3.8 FlashAPI71Needs more work Gemini 3.5 Flash-LiteGemini52Needs more work

Gemini 3.5 Flash-LiteGemini

Needs more than a quick edit5 of 11 checks passed

The output incorrectly marks Android 13 as go with conditions despite failing the error-free sync criterion, and lacks post-launch monitoring thresholds, so a PM would need to re-check the criteria and add missing guardrails.

Output

To: Leadership Team From: PM, Lumen Mobile Apps Date: March 28, 2024 Subject: Recommendation: Offline Sync 100% Rollout (Friday Launch Review)

---

Executive Summary

We have completed the 14-day, 25% rollout of Offline Sync across iOS and Android. Based on the data, incident logs, and agreed launch criteria, my recommendation is a segmented decision for Monday’s full rollout:

iOS 17+: GO. All metrics meet or exceed criteria with zero critical incidents.
Android 14+ and Android 13: GO WITH CONDITIONS. Stability is acceptable, but minor degradation requires a hotfix commitment before 100% exposure.
Android 12 and older: NO-GO. Fails multiple hard launch criteria, including data loss incidents. We must exclude these OS versions from the 100% rollout.

---

Launch Criteria Status Matrix

Platform / OS	1. Crash-Free (Max -0.2% vs control)	2. Conflict Tickets (<2 / 1k users)	3. Error-Free Sync (≥99%)	4. No Data Loss	Final Recommendation & Rationale
:---	:---:	:---:	:---:	:---:	:---
iOS 17+	GO<br>(99.58% vs 99.62% control; Δ -0.04)	GO<br>(0.6 / 1k)	GO<br>(99.4%)	GO<br>(0 incidents)	GO. Performs well within tolerance. One unverified slow-sync report does not block.
Android 14+	GO<br>(99.49% vs 99.55% control; Δ -0.06)	GO<br>(0.8 / 1k)	GO<br>(99.2%)	GO<br>(0 incidents)	GO WITH CONDITIONS. Stable, but error-free sync is close to the 99% floor. Ship with expedited telemetry monitoring.
Android 13	GO<br>(99.31% vs 99.40% control; Δ -0.09)	GO<br>(1.1 / 1k)	GO WITH CONDITIONS<br>(98.9%)	GO<br>(0 incidents)	GO WITH CONDITIONS. Sync error rate dips slightly below 99% (98.9%). Acceptable for rollout only alongside a targeted patch.
Android 12 and older	NO-GO<br>(98.71% vs 99.38% control; Δ -0.67)	NO-GO<br>(4.9 / 1k)	NO-GO<br>(96.1%)	NO-GO<br>(2 data loss incidents)	NO-GO. Fails all criteria. Legacy storage APIs caused two overwrites. Exclude from 100% release.

---

Key Findings & Rationale

iOS is Fully Vetted: iOS users experience virtually identical stability to the control group. The single support ticket regarding slow sync was an isolated, non-reproducible event.
Modern Android is Stable, but Needs Polish: Android 13 and 14+ show minor dips in error-free sync rates (98.9% and 99.2%, respectively) and slight increases in conflict tickets, but both remain safely within agreed thresholds and exhibit zero data loss.
Legacy Android is a Risk: Android 12 and older failed Criterion 1 (crash-free sessions dropped by 0.67%), Criterion 2 (4.9 conflict tickets per 1k users), Criterion 3 (96.1% sync success), and violated Criterion 4 (two confirmed data loss incidents due to legacy storage API conflicts). Although engineering successfully recovered the notes from backups, data loss is a hard stop for a 100% rollout.

---

Action Plan for Monday’s Launch

1. Modify the Rollout Scope: * Push 100% of Offline Sync to iOS 17+ and Android 13+. * Halt the rollout for Android 12 and older, keeping these users on the standard online-only experience until a refactored storage handler is built. 2. Marketing Alignment: Proceed with Monday’s email blast to all users, with a minor internal note to customer support regarding known edge cases on legacy Android devices. 3. Engineering Post-Launch Focus: Prioritize a patch for legacy Android storage APIs to bring older devices up to launch standard for a secondary release next quarter.

Grades and run details

Decision model 50 · LLM judge 58

Decision model checks

failUses the supplied evidence correctly16%
partialAddresses the actual decision37%
failRespects explicit constraints35%
partialIdentifies material uncertainty87%
partialAvoids unsupported claims30%
partialProduces the required deliverable44%
partialChecks every criterion for every platform25%
passTreats the overwrites as data loss96%
partialLimits the downside of being wrong96%
passGets the base of every number right49%
partialJudges the launch against a baseline set now82%

Run

Run: #1
Time to output: 5 s
Submitted: 5 Oct 2026

GPT-6 AstraChatGPT

Usable with a quick edit10 of 11 checks passed

The output is clear, complete, and actionable; the missing explicit review point is a small omission that a PM could add with a quick edit without rechecking evidence or redoing reasoning.

Output

Recommendation for Friday

No-go for a blanket 100% launch. Approve a platform-gated launch for iOS 17+ and Android 14+, provided eligibility can be enforced reliably. Hold Android 13, stop expansion on Android 12 and older, and do not launch on untested platforms.

Replace Monday’s all-user email with a targeted announcement to eligible users only. If targeting or platform gating cannot be verified before send time, postpone the email.

Launch-criteria assessment

Crash-free changes below are percentage-point differences versus each platform’s control. The agreed limit is a decline of no more than 0.20 points.

Platform	Crash-free sessions	Conflict tickets: <2/1,000	Notes synced without error: ≥99%	No data loss incidents	Platform decision
iOS 17+	Go: 99.58%; −0.04 points	Go: 0.6	Go: 99.4%	Go: none reported	Go with conditions: passes all measured criteria; launch with eligibility gating and monitoring.
Android 14+	Go: 99.49%; −0.06 points	Go: 0.8	Go: 99.2%	Go: none reported	Go with conditions: passes all measured criteria; launch with eligibility gating and monitoring.
Android 13	Go: 99.31%; −0.09 points	Go: 1.1	No-go: 98.9%, below the 99% minimum	Go: none reported	No-go: sync reliability misses the agreed threshold.
Android 12 and older	No-go: 98.71%; −0.67 points	No-go: 4.9	No-go: 96.1%	No-go: two overwrite incidents on Android 11	No-go: fails all four criteria.
Web / iOS below 17, if supported	No-go: no evidence supplied	No-go: no evidence supplied	No-go: no evidence supplied	No-go: not established by supplied evidence	No-go pending validation: absence of rollout data is not a pass.

The recovered Android 11 edits still count as data-loss incidents for this launch gate: users’ edits were overwritten, and recovery required backups. Recovery mitigated the harm; it does not erase the incidents. Nor should Android 11’s identified root cause be used to clear the rest of the “Android 12 and older” cohort without separate evidence.

Conditions and next steps

Before expanding eligible cohorts: Engineering must verify OS-level gating, rollback controls, and platform-specific monitoring. Increase exposure in monitored steps toward 100% of eligible users, pausing if a launch criterion is breached. Investigate the unreproduced iOS slow-sync report; it is not currently evidence of a failed criterion.
Android 13: Diagnose the sync-error shortfall, ship a fix, and repeat validation against the same ≥99% threshold. Do not round 98.9% up or waive the criterion to meet the email date.
Android 12 and older: Halt expansion and safely disable affected functionality where feasible, preserving pending offline edits before any rollback. Fix the older-storage-API issue, test conflict recovery and data integrity, and rerun a controlled rollout demonstrating all four criteria.
Marketing and support: Clearly state supported versions and avoid implying universal availability. Give support a recovery and escalation playbook.

These results cover 14 days at 25% exposure, not proof of risk-free operation at full scale. The defensible decision is a constrained launch where the evidence passes—not a universal launch driven by Monday’s booking.

Grades and run details

Decision model 82 · LLM judge 92

Decision model checks

passUses the supplied evidence correctly38%
passAddresses the actual decision80%
partialRespects explicit constraints30%
passIdentifies material uncertainty89%
passAvoids unsupported claims47%
passProduces the required deliverable70%
partialChecks every criterion for every platform66%
passTreats the overwrites as data loss100%
partialLimits the downside of being wrong56%
passGets the base of every number right81%
partialJudges the launch against a baseline set now26%

Run

Run: #1
Time to output: 16 s
Submitted: 5 Oct 2026

Gemini 3.8 FlashAPI

Needs more than a quick edit8 of 11 checks passed

The recommendation lacks post-launch monitoring thresholds and a baseline review for the iOS and Android 14+ rollout; a PM would need to add those before acting.

Output

To: Executive Team & Launch Review Committee From: Mobile Product Manager, Lumen Date: October 25, 2024 Subject: Launch Review: Offline Sync Recommendation (Mobile)

---

Recommendation: Go with Conditions (Segmented Launch)

Proceed with a 100% rollout for iOS 17+ and Android 14+, but kill-switch/disable offline sync for Android 12 and older. Keep Android 13 held at 25% through Monday pending review.

Marketing’s Monday email blast must be updated to segment the audience: promote the feature only to eligible platforms, rather than blasting all users with an unsegmented announcement.

---

Launch Criteria Evaluation

Launch Criterion	iOS 17+	Android 14+	Android 13	Android 12 & older
:---	:---:	:---:	:---:	:---:
1. Crash-free sessions<br>(≤ 0.2% drop vs control)	PASS<br>-0.04% (99.58% vs 99.62%)	PASS<br>-0.06% (99.49% vs 99.55%)	PASS<br>-0.09% (99.31% vs 99.40%)	FAIL<br>-0.67% (98.71% vs 99.38%)
2. Sync-conflict tickets<br>(< 2.0 per 1k users)	PASS<br>0.6 / 1k	PASS<br>0.8 / 1k	PASS<br>1.1 / 1k	FAIL<br>4.9 / 1k
3. Clean sync rate<br>(≥ 99.0% without error)	PASS<br>99.4%	PASS<br>99.2%	FAIL<br>98.9% (-0.1%)	FAIL<br>96.1%
4. No data loss incidents	PASS<br>0 incidents	PASS<br>0 incidents	PASS<br>0 incidents	FAIL<br>2 incidents (Android 11)
Platform Decision	GO (100%)	GO (100%)	HOLD (25%)	NO-GO (0%)

---

Key Findings & Justification

1. iOS 17+ and Android 14+ (80% of our mobile base) are fully launch-ready. Both platforms exceed every bar comfortably. Crash impact is negligible (-0.04% to -0.06%), error-free sync rates exceed 99.2%, conflict tickets are low, and zero data-loss events occurred.

2. Android 12 and older is an unacceptable risk (Hard NO-GO). Android 12 and older failed every single metric. Most critically, it caused two data-loss incidents on Android 11, where conflicting edits were overwritten due to a deprecated storage API. While backups saved the content, our core value proposition as a collaborative team notes app relies on zero data loss. Coupled with high ticket volumes (4.9 / 1k) and a significant crash-rate drop (-0.67%), the offline sync flag must be completely turned off for Android ≤12 until the legacy storage integration is re-architected.

3. Android 13 is on the cusp (HOLD at 25%). Android 13 passed stability, conflict rate, and data loss criteria, but missed the clean sync rate threshold by 0.1% (98.9% vs. 99.0%). Because it remains stable and safe from data loss, we should hold at 25% over the weekend. Engineering will analyze non-fatal sync logs on Friday. If the 0.1% variance is cosmetic (e.g., immediate automatic retries), we can bump to 100% on Monday; otherwise, we hold until patched.

---

Conditions for Monday’s Launch

1. Targeted Feature Flag Activation (Friday COB): * Enable offline sync to 100% on iOS 17+ and Android 14+. * Maintain Android 13 at 25% treatment. * Remote-kill the offline sync flag (0%) for all clients on Android 12 and below.

2. Marketing Campaign Segmentation (Friday / Monday Morning): * Do not send an unsegmented blast. Promoting offline sync to users on older Android devices who cannot use it—or worse, who experience bugs—will spike support volume and erode trust. * Plan A (Recommended): Segment the Monday email list. Send the offline sync announcement only to iOS and Android 14+ active users. * Plan B (Fallback): If audience segmentation cannot be configured before Monday, amend the copy to state: "Now rolling out to iOS and Android 14+ (older Android versions coming soon)."

3. Engineering Follow-Up: * Triage Android 13 sync error logs by end-of-day Friday. * File remediation sprint for Android 11/12 legacy storage API integration.

Grades and run details

Decision model 73 · LLM judge 75

Decision model checks

failUses the supplied evidence correctly18%
passAddresses the actual decision64%
partialRespects explicit constraints17%
passIdentifies material uncertainty87%
partialAvoids unsupported claims51%
passProduces the required deliverable61%
partialChecks every criterion for every platform76%
passTreats the overwrites as data loss100%
partialLimits the downside of being wrong46%
passGets the base of every number right47%
passJudges the launch against a baseline set now68%

Run

Run: #1
API response time: 22 s
Submitted: 6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 6

Uses the supplied evidence correctlyMixedRightMixed

Gemini 3.5 Flash-Lite · Gemini

All factual claims about current performance, numbers, and incidents are taken directly from the supplied rollout data and incident log, with no invented facts.

GPT-6 Astra · ChatGPT

All factual claims about the current situation are taken directly from the supplied rollout data, incidents log, and launch criteria, with no invented numbers or events.

Gemini 3.8 Flash · API

Every fact about the current situation is directly from the supplied context or follows from arithmetic; no invented numbers or events.

Respects explicit constraintsMixedRightRight

Gemini 3.5 Flash-Lite · Gemini

The output is a recommendation memo under 700 words, includes the required table with go/no-go/go with conditions per platform, and addresses the leadership team as the reader.

GPT-6 Astra · ChatGPT

The output is a recommendation under 700 words, includes the required table with go/no-go/go with conditions per platform, and addresses the launch review decision.

Gemini 3.8 Flash · API

The output is a recommendation under 700 words, includes the required table with go/no-go/conditions per platform and criterion, and does not propose a 100% rollout to every platform.

Identifies material uncertaintyWrongRightRight

Gemini 3.5 Flash-Lite · Gemini

Does not name specific unknowns that could change the decision, does not bound them, and does not say how they would be resolved or what post-launch signal would trigger a pause or rollback.

GPT-6 Astra · ChatGPT

It names the unreproduced iOS slow-sync report, the need to verify platform gating, and the Android 13 sync-error shortfall as unknowns, and says how each would be resolved or would change the call.

Gemini 3.8 Flash · API

Identifies the specific uncertainty on Android 13 (cosmetic vs real sync error) and says how it will be resolved (log analysis Friday) and what each outcome would mean.

Checks every criterion for every platformWrongRightRight

Gemini 3.5 Flash-Lite · Gemini

Android 13's error-free sync is 98.9%, below the 99% threshold, yet the table marks it as 'GO WITH CONDITIONS' instead of 'NO-GO' for that criterion, misapplying the hard launch criteria.

GPT-6 Astra · ChatGPT

The table correctly computes each criterion for every platform against its control and threshold, and correctly identifies Android 12 and older as failing (and Android 13 as failing the sync-error criterion).

Gemini 3.8 Flash · API

The table evaluates each criterion per platform correctly, with the correct figures, and identifies Android 12 and older as failing; Android 13 correctly fails the sync-error criterion.

Limits the downside of being wrongWrongRightWrong

Gemini 3.5 Flash-Lite · Gemini

Does not name a post-launch signal, a threshold that would pause or roll back, or flag irreversible actions; only mentions 'expedited telemetry monitoring' without specifics.

GPT-6 Astra · ChatGPT

It names the post-launch signal (launch criteria), the threshold (breach of any criterion), the action (pause), and flags the irreversible email by recommending targeting or postponement.

Gemini 3.8 Flash · API

Does not name the signals, thresholds or triggers that would pause or roll back the 100% rollout on iOS 17+ and Android 14+; only addresses monitoring for the Android 13 hold.

Judges the launch against a baseline set nowWrongWrongMixed

Gemini 3.5 Flash-Lite · Gemini

Does not name a success metric with its current baseline from the data, nor a review point with what each result would lead to.

GPT-6 Astra · ChatGPT

The output does not name a specific success metric with its baseline from the material, nor a scheduled review point with what each result would lead to; it only mentions pausing on breach without a defined review cadence.

Gemini 3.8 Flash · API

No mention of a specific success metric and its baseline to judge the launch, nor a review point with what each result would mean for the full rollout.

All got right 5

Addresses the actual decisionRightRightRight

Gemini 3.5 Flash-Lite · Gemini

Commits early to a segmented go/no-go call per platform, framed for the leadership team, and states that a refactored storage handler would change the no-go for Android 12 and older.

GPT-6 Astra · ChatGPT

The output commits early to a clear call (no blanket launch, platform-gated go for iOS 17+ and Android 14+, hold Android 13, stop Android 12 and older) and states what would change it (gating verification, fix and re-validation).

Gemini 3.8 Flash · API

Commits to a segmented 'Go with Conditions' answer, frames it for the executive team, and specifies what would change the Android 13 hold.

Avoids unsupported claimsRightRightRight

Gemini 3.5 Flash-Lite · Gemini

Interpretive statements like 'iOS is Fully Vetted' are clearly opinion, and causal claims (legacy storage API) are directly supported by the incident log.

GPT-6 Astra · ChatGPT

Interpretations (e.g., the overwrites counting as data loss) are clearly presented as judgments based on the criteria, not as established facts, and no unsupported causal claims are made.

Gemini 3.8 Flash · API

The Android 13 variance hypothesis is clearly labelled ('If … cosmetic'), and all confident claims are backed by the evidence.

Produces the required deliverableRightRightRight

Gemini 3.5 Flash-Lite · Gemini

The memo is complete, within word limit, contains the required table, and a leadership team could act on the segmented recommendation with light edits.

GPT-6 Astra · ChatGPT

The output is a complete recommendation with a table, within the word limit, and a PM could act on it with only light edits.

Gemini 3.8 Flash · API

A recommendation with a complete table, under 700 words, addressed to the right audience, and actionable on Monday with only light edits.

Treats the overwrites as data lossRightRightRight

Gemini 3.5 Flash-Lite · Gemini

Explicitly treats the two overwritten notes on Android 11 as a breach of the no-data-loss criterion and makes fixing the storage API a condition for that platform.

GPT-6 Astra · ChatGPT

The output explicitly treats the Android 11 overwrites as a breach of the no-data-loss criterion and makes fixing the storage-API issue a condition for that platform.

Gemini 3.8 Flash · API

Treats the overwritten notes as a no-data-loss breach, marks it as FAIL for Android 12 and older, and makes fixing the storage API a condition for that platform.

Gets the base of every number rightRightRightRight

Gemini 3.5 Flash-Lite · Gemini

All percentage differences are computed correctly from the control and offline-sync values in the CSV, and the base (percentage points) is clear.

GPT-6 Astra · ChatGPT

All percentage-point differences are computed correctly from the control and offline-sync values, and the base (percentage points vs control) is stated clearly.

Gemini 3.8 Flash · API

All percentage differences, ticket rates, and the 80% user base figure are derived correctly from the supplied data.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 63% of 68 checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	Sonnet 5.5withAPI	87.8	100.0	2	None
2	Haiku 5.5withAPI	82.5	100.0	2	None
3	GPT-6.1 SolwithAPI	86.0	95.8	2	None
4	Opus 5.5withClaude	84.3	96.7	2	None
5	GPT-6 AstrawithChatGPT	83.8	92.5	2	None
6	GPT-6 LunawithAPI	74.8	80.8	2	None
7	Gemini 3.8 FlashwithAPI	64.9	67.5	2	1 capped
8	Gemini 3.5 Flash-LitewithGemini	48.2	59.2	2	None

What you get for the cost

Haiku 5.5 · API scores within 2 points of Sonnet 5.5 · API at about a tenth of the cost.

Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)At least one critical failure

Task score and relative cost by setup
Setup	Task score	Relative cost (cheapest = 1)
GPT-6 Luna · API	77	1.0×
Gemini 3.5 Flash-Lite · Gemini	52	2.7× (partly estimated)
Haiku 5.5 · API	92	2.8×
Gemini 3.8 Flash · API	55	11×
GPT-6.1 Sol · API	91	12×
Sonnet 5.5 · API	94	37×
GPT-6 Astra · ChatGPT	89	46× (partly estimated)
Opus 5.5 · Claude	91	59× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Deciding whether a feature ships on the planned date, and on what conditions.

Why it matters

Launch meetings reward optimism. A good call checks each agreed criterion, weighs a real risk against a real gain, and says exactly what would change the answer. A weak one rubber-stamps the launch or blocks it on noise.

What good looks like

Checks each agreed criterion against the evidence
Makes one clear call, with conditions if needed
Separates risks that block launch from risks that can be managed
Says what would change the call
Says how to limit the damage if the call is wrong

Deliberately not measured

Rollout engineering detail
Project-plan formatting

Capability tested

Deciding against agreed launch criteria

The failure we’re looking for

Rubber-stamps a launch that misses an agreed bar, or blocks it on noise

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

A launch memo from supplied evidence · Staff level: a regional call from an attached data workbook

What AI gets right here, and what you’ll still have to catch

Reliably right

Where it slips

How it’s graded

Limits the downside of being wrong

Judges the launch against a baseline set now

Plus our standard checks

The tasks

The outputs

Output

Executive Summary

Launch Criteria Status Matrix

Key Findings & Rationale

Action Plan for Monday’s Launch

Grades and run details

Decision model checks

Run

Output

Recommendation for Friday

Launch-criteria assessment

Conditions and next steps

Grades and run details

Decision model checks

Run

Output

Recommendation: Go with Conditions (Segmented Launch)

Launch Criteria Evaluation

Key Findings & Justification

Conditions for Monday’s Launch

Grades and run details

Decision model checks

Run

Check by check

Where they differ 6

All got right 5

Results

What you get for the cost

About the task

The PM job

Why it matters

What good looks like

Deliberately not measured