Changelog

Task pages plot each setup's score against what an output costs at list price, as a multiple of the cheapest setup, and name the best value: the cheapest setup within 5 points of the leader at half its cost or less. Cost never changes a score.

Each of 15 task types gets a new Staff-level task built on files: CSV exports, ticket logs, scorecards, interview notes and drafts, with the traps in the data rather than in a summary (a metric redefined mid-series, one customer behind most requests, a sample ratio mismatch, a market size that doesn't survive the cohorts). Frontier models were acing the tasks they replace. Each new task runs alongside the old one until every model has been graded on it, then the old task retires, so scores on these task types will move over the next day.

Every task type is now graded on how world-class product and growth leaders work, as they've described it on Lenny's Podcast, with the leader credited by name beside each check. Fifteen new checks join, and task types are grouped into five domains: Product Craft, Growth, Metrics & Experimentation, Strategy and Leadership. A task type now also needs three reviewed verdicts on every one of its checks before it counts as calibrated, so most are provisional until the new checks are reviewed.

Two new checks: separating the hard-to-reverse decisions leaders need to make from the ones the team owns (Geoff Charles), and reporting progress as outcomes and what was learned (Melissa Perri and Christian Idiodi).

The Combined score is now the share of checks an output passes, averaged across the decision model and the LLM judge, and capped at 40 for a critical failure. Our PM no longer scores outputs; they check the checkers. A task type counts as calibrated once the graders agree with them on at least 80% of checks.

“Addresses the actual decision” now only passes when an output commits to one answer and says what would change it. Five task types also gained a check drawn from the fixes reviewers kept making, such as getting the base of every number right, or saying how many sources support each finding.

Each task type now has one check for the sharpest thing experienced product leaders say about doing that job well, such as diagnosis before prescription for strategy, or trusting the data before reading an experiment. Every output was re-graded with it.

Each of the three task types swapped a task for a much harder one: a long, interlocking evidence pack, a slide deck in fixed brand colours, and a data workbook to work through. Scores on these task types now rest partly on the new tasks.

Two Operate task types join: writing a stakeholder update and making a launch call. “Sticking to the plan” leaves, because judging it fairly needs a real codebase, and every task type already penalises work nobody asked for.

Every written output is also graded by an LLM judge from a model family we don't test, so it isn't marking its own homework. It checks each claim against the brief, then answers every check yes or no.