Tasks / Product Craft

Make the launch call

Can the model make a clear go / no-go call from mixed launch evidence, checked against the criteria agreed up front?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 69% were usable with at most a quick edit.

Reliably right

Finds the Scotland breach100% pass
Finds Scotland's refund breach (+66% overall, +143% since 15 Oct), ties it to the supplier change, notes acceptance dip, and holds Scotland.
Sonnet 5.5 · API · Go/no-go for AI substitutions, from the rollout data
Treats the overwrites as data loss100% pass
Treats the overwritten notes on Android 11 as a breach of the no-data-loss criterion and makes fixing the storage-API issue a condition for that platform.
Opus 5.5 · Claude · Offline, on older phones
Addresses the actual decision98% pass
Commits to a clear no-go for 2 Nov, conditional go from 4 Nov for four regions, and no-go for Scotland, with conditions that would change it.
Sonnet 5.5 · API · Go/no-go for AI substitutions, from the rollout data

Where it slips

Limits the downside of being wrong63% pass
It does not name a specific post-launch signal, a threshold that would trigger pause or rollback, or flag any irreversible actions.
GPT-6 Luna · API · Go/no-go for AI substitutions, from the rollout data
Judges the launch against a baseline set now66% pass
Does not name a specific review point or when the team will review results to decide next steps; only gives a pause trigger without a scheduled evaluation.
GPT-6.1 Sol · API · Offline, on older phones
Checks every criterion for every platform69% pass
Android 13's error-free sync is 98.9%, below the 99% threshold, yet the table marks it as 'GO WITH CONDITIONS' instead of 'NO-GO' for that criterion, misapplying the hard launch criteria.
Gemini 3.5 Flash-Lite · Gemini · Offline, on older phones

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

Limits the downside of being wrong
Does the call say how exposure is limited if it's wrong: what is watched after launch, the threshold that would pause or roll back, and which parts can't be undone?
Passes when Names the post-launch signal, a threshold or trigger, and the action, and flags anything irreversible (customer communications, pricing, migrations).
Karri SaarinenLenny’s Podcast
Judges the launch against a baseline set now
Does the call name the metric and the current baseline the launch will be judged against, and when the team will review the result to decide what's next?
Passes when Names the success metric, its baseline from the material, and a review point with what each result would lead to.
Todd JacksonLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for Lumen's mobile apps. The launch review for offline sync is on Friday, and marketing has an email booked for Monday to every user. Write your recommendation, in no more than 700 words, with a table checking each launch criterion for each platform: go, no-go or go with conditions, and why. The rollout data, the criteria and the incident log are attached.

What the model was given5 items: About Lumen, launch_criteria.md, rollout_25pct_14_days.csv, incidents.md, the_decision.md

About LumenA notes app for teams, on iOS, Android and the web. Offline sync lets people edit notes without a connection and merges changes when they reconnect.

launch_criteria.md6 lines · Download

# Launch criteria (agreed before the rollout)

1. Crash-free sessions no more than 0.2 points below control, per platform.
2. Sync-conflict tickets under 2 per 1,000 users.
3. Notes synced without error at least 99%.
4. No data loss incidents.

rollout_25pct_14_days.csv8 rows · Download

platform,arm,users,crash_free_sessions_pct,sync_conflict_tickets_per_1000_users,notes_synced_without_error_pct
iOS 17+,control,123600,99.62,0.3,99.5
iOS 17+,offline sync,41200,99.58,0.6,99.4
Android 14+,control,68400,99.55,0.3,99.5
Android 14+,offline sync,22800,99.49,0.8,99.2
Android 13,control,27300,99.4,0.3,99.5
Android 13,offline sync,9100,99.31,1.1,98.9
…

incidents.md4 lines · Download

# Incidents during the 25% rollout (14 days)

- 2 reports of a note's edits being overwritten after a conflict, both on Android 11; engineering recovered both from backups. Root cause: an older Android storage API.
- 1 report of slow sync on iOS, not reproduced.

the_decision.md1 lines · Download

The launch review is Friday: roll offline sync out to 100%, or not. Marketing has a launch email booked for Monday to every user.

What a strong answer doesThe answer key the graders mark against

Checks each criterion per platform. iOS 17+, Android 14+ and Android 13 pass: crash-free sessions within 0.2 points of control (iOS -0.04, Android 14+ -0.06, Android 13 -0.09), sync-conflict tickets under 2 per 1,000, notes synced without error at least 99%. Android 12 and older fails three criteria: crash-free down 0.67 points against control, 4.9 conflict tickets per 1,000, 96.1% synced without error; and the two overwritten-note incidents were on Android 11, which breaks 'no data loss' (both recovered from backup, root cause an older storage API). Recommends go for iOS, Android 13 and newer, and no-go for Android 12 and older until the storage issue is fixed and re-tested; the Monday email should only go to users on supported versions, or wait. States what's monitored after launch and the threshold that would pause it.

Critical failures (cap the score at 40)

Recommends rolling out to 100% on every platform

Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Opus 5.5Claude94Usable with a quick edit Sonnet 5.5API94Usable with a quick edit Haiku 5.5API94Usable with a quick edit GPT-6.1 SolAPI90Usable with a quick edit GPT-6 AstraChatGPT88Usable with a quick edit GPT-6 LunaAPI75Needs more work Gemini 3.8 FlashAPI71Needs more work Gemini 3.5 Flash-LiteGemini52Needs more work

Gemini 3.5 Flash-LiteGemini

Needs more than a quick edit5 of 11 checks passed

The output incorrectly marks Android 13 as go with conditions despite failing the error-free sync criterion, and lacks post-launch monitoring thresholds, so a PM would need to re-check the criteria and add missing guardrails.

Output

To: Leadership Team From: PM, Lumen Mobile Apps Date: March 28, 2024 Subject: Recommendation: Offline Sync 100% Rollout (Friday Launch Review)

---

Executive Summary

We have completed the 14-day, 25% rollout of Offline Sync across iOS and Android. Based on the data, incident logs, and agreed launch criteria, my recommendation is a segmented decision for Monday’s full rollout:

iOS 17+: GO. All metrics meet or exceed criteria with zero critical incidents.
Android 14+ and Android 13: GO WITH CONDITIONS. Stability is acceptable, but minor degradation requires a hotfix commitment before 100% exposure.
Android 12 and older: NO-GO. Fails multiple hard launch criteria, including data loss incidents. We must exclude these OS versions from the 100% rollout.

---

Launch Criteria Status Matrix

Platform / OS	1. Crash-Free (Max -0.2% vs control)	2. Conflict Tickets (<2 / 1k users)	3. Error-Free Sync (≥99%)	4. No Data Loss	Final Recommendation & Rationale
:---	:---:	:---:	:---:	:---:	:---
iOS 17+	GO<br>(99.58% vs 99.62% control; Δ -0.04)	GO<br>(0.6 / 1k)	GO<br>(99.4%)	GO<br>(0 incidents)	GO. Performs well within tolerance. One unverified slow-sync report does not block.
Android 14+	GO<br>(99.49% vs 99.55% control; Δ -0.06)	GO<br>(0.8 / 1k)	GO<br>(99.2%)	GO<br>(0 incidents)	GO WITH CONDITIONS. Stable, but error-free sync is close to the 99% floor. Ship with expedited telemetry monitoring.
Android 13	GO<br>(99.31% vs 99.40% control; Δ -0.09)	GO<br>(1.1 / 1k)	GO WITH CONDITIONS<br>(98.9%)	GO<br>(0 incidents)	GO WITH CONDITIONS. Sync error rate dips slightly below 99% (98.9%). Acceptable for rollout only alongside a targeted patch.
Android 12 and older	NO-GO<br>(98.71% vs 99.38% control; Δ -0.67)	NO-GO<br>(4.9 / 1k)	NO-GO<br>(96.1%)	NO-GO<br>(2 data loss incidents)	NO-GO. Fails all criteria. Legacy storage APIs caused two overwrites. Exclude from 100% release.

---

Key Findings & Rationale

iOS is Fully Vetted: iOS users experience virtually identical stability to the control group. The single support ticket regarding slow sync was an isolated, non-reproducible event.
Modern Android is Stable, but Needs Polish: Android 13 and 14+ show minor dips in error-free sync rates (98.9% and 99.2%, respectively) and slight increases in conflict tickets, but both remain safely within agreed thresholds and exhibit zero data loss.
Legacy Android is a Risk: Android 12 and older failed Criterion 1 (crash-free sessions dropped by 0.67%), Criterion 2 (4.9 conflict tickets per 1k users), Criterion 3 (96.1% sync success), and violated Criterion 4 (two confirmed data loss incidents due to legacy storage API conflicts). Although engineering successfully recovered the notes from backups, data loss is a hard stop for a 100% rollout.

---

Action Plan for Monday’s Launch

1. Modify the Rollout Scope: * Push 100% of Offline Sync to iOS 17+ and Android 13+. * Halt the rollout for Android 12 and older, keeping these users on the standard online-only experience until a refactored storage handler is built. 2. Marketing Alignment: Proceed with Monday’s email blast to all users, with a minor internal note to customer support regarding known edge cases on legacy Android devices. 3. Engineering Post-Launch Focus: Prioritize a patch for legacy Android storage APIs to bring older devices up to launch standard for a secondary release next quarter.

Grades and run details

Decision model 50 · LLM judge 58

Decision model checks

failUses the supplied evidence correctly16%
partialAddresses the actual decision37%
failRespects explicit constraints35%
partialIdentifies material uncertainty87%
partialAvoids unsupported claims30%
partialProduces the required deliverable44%
partialChecks every criterion for every platform25%
passTreats the overwrites as data loss96%
partialLimits the downside of being wrong96%
passGets the base of every number right49%
partialJudges the launch against a baseline set now82%

Run

Run: #1
Time to output: 5 s
Submitted: 5 Oct 2026

GPT-6 LunaAPI

Needs more than a quick edit9 of 11 checks passed

Missing a post-launch monitoring plan with thresholds and a baseline for success, which a PM would need to add before acting.

Output

Recommendation

Do not roll offline sync out to every user or send Monday’s email to every user as a general launch announcement. The agreed criteria are not met on Android 13 or Android 12 and older, and the Android 11 overwrite reports fail the no-data-loss gate even though the notes were recovered from backups.

Approve rollout to 100% only for iOS 17+ and Android 14+, where all four criteria pass. Keep Android 13 and older gated until they meet the criteria and the Android 11 storage issue is fixed and validated. Change Monday’s email to target eligible users, or postpone it; don’t imply the feature is available to everyone. Monitor the unreproduced iOS slow-sync report.

Platform / cohort	Decision	Crash-free sessions	Conflict tickets	Notes synced without error	No data loss
iOS 17+	Go	Go — 0.04 percentage points below control; within 0.2.	Go — 0.6/1,000; under 2.	Go — 99.4%; at least 99%.	Go — no data-loss incident reported.
Android 14+	Go	Go — 0.06 points below control; within 0.2.	Go — 0.8/1,000; under 2.	Go — 99.2%; at least 99%.	Go — no data-loss incident reported.
Android 13	Go with conditions — hold rollout until notes-sync criterion passes.	Go — 0.09 points below control; within 0.2.	Go — 1.1/1,000; under 2.	No-go — 98.9%; below 99%.	Go — no data-loss incident reported.
Android 12 and older	No-go — keep blocked; investigate and validate by OS version.	No-go — 0.67 points below control; exceeds 0.2.	No-go — 4.9/1,000; exceeds 2.	No-go — 96.1%; below 99%.	No-go — the cohort includes Android 11, where two overwritten edits were reported. Recovery does not erase the incident.

The Android 12-and-older metrics are grouped, so they do not establish that each OS version performs equally; assess Android 12 separately before reconsidering it.

Grades and run details

Decision model 82 · LLM judge 75

Decision model checks

passUses the supplied evidence correctly67%
passAddresses the actual decision79%
passRespects explicit constraints34%
partialIdentifies material uncertainty36%
passAvoids unsupported claims22%
passProduces the required deliverable43%
partialChecks every criterion for every platform45%
passTreats the overwrites as data loss100%
partialLimits the downside of being wrong98%
passGets the base of every number right66%
partialJudges the launch against a baseline set now82%

Run

Run: #1
API response time: 21 s
Submitted: 6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.