Humanity’s Last PRD
Latest drop: Haiku 5.5 · 8 Oct
Tasks / Product Craft

One-shot prototype

Can the configuration produce a working, constraint-compliant prototype in one attempt?

Task type v1.2 · 2 tasksLast changed 27 Sept 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models.

Reliably right

  1. Avoids template layout and copy tells100% pass
  2. Shows the no-slots state100% pass
  3. Respects explicit constraints88% pass

Where it slips

  1. Says what it tests and what's faked44% pass
  2. Avoids the generic AI palette75% pass
  3. Stays in scope78% pass

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Says what it tests and what's faked

    Does the reply that comes with the prototype say what question or assumption it is meant to test, and which parts are faked, hard-coded or missing?

    Passes when Names what the prototype is for, and lists the stubbed or hard-coded parts (data, sign-in, saving and the like).

    Karri SaarinenLenny’s Podcast

Plus our standard checks

· and 4 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

Build a clickable prototype of a flow where an employee captures a receipt, checks the amount the app extracted, corrects it, and saves the expense. Deliver a single self-contained HTML file that opens straight into the flow in a browser: no design canvas, artboard or "play" step. Use made-up data.

What the model was given2 items: Scenario, Constraints
ScenarioAlex Morgan is submitting a taxi receipt from 14 September for £38.40. The app's automatic extraction misreads the amount as £83.40.
ConstraintsDesktop web. Receipt capture is simulated: a control such as "Use sample receipt" stands in for the camera or file upload, with no real upload needed. Show the extracted amount and let the employee correct it before saving; the saved expense shows the corrected amount.
What a strong answer doesThe answer key the graders mark against

Capture (simulated), a visible extracted amount, an easy correction, and a saved expense showing the corrected amount.

Critical failures (cap the score at 40)
  • Core interaction broken
Task

Mid-level · v1.1

The outputs

8 setups answered this brief.

Sonnet 5.5API100Judged in the browserGPT-6.1 SolAPI92Judged in the browserHaiku 5.5API92Judged in the browserGPT-6 LunaAPI83Judged in the browserOpus 5.5Claude75Judged in the browserGPT-6 AstraChatGPT75Judged in the browserGemini 3.8 FlashAPI58Judged in the browserGemini 3.5 Flash-LiteGemini42Judged in the browser
Haiku 5.5API

5 of 6 checks passed

Output

Open the prototypeThe file the model delivered, in a new tab

The prototype opens on the capture screen. Clicking "Use sample receipt" (or "Choose file", which loads the same sample) moves to the check screen, where the app shows £83.40 and the receipt on the left shows £38.40. The employee edits the amount, must tick the confirmation box, then saves. The saved expense and the expense list show the corrected £38.40 and note the original £83.40 reading.

Grades and run details

Decision model 92
Decision model checks
  • passStays in scope54%
  • partialSays what it tests and what's faked43%
  • passRespects explicit constraintsby hand100%
  • passCore interaction worksby hand100%
  • passAvoids the generic AI paletteby hand100%
  • passAvoids template layout and copy tellsby hand100%
Artefacts
Run
Run
#1
API response time
48 s
Submitted
8 Oct 2026

Check by check

Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Mixed 1

Says what it tests and what's fakedMixed

No reason given.

Got right 5

Respects explicit constraintsRight

No reason given.

Core interaction worksRight

No reason given.

Stays in scopeRight

No reason given.

Avoids the generic AI paletteRight

No reason given.

Avoids template layout and copy tellsRight

No reason given.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 78% of 78 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Sonnet 5.5API92.9–2None
2Haiku 5.5API92.3–2None
3GPT-6.1 SolAPI88.7–2None
4GPT-6 LunaAPI84.5–2None
5Opus 5.5Claude76.8–2None
6Gemini 3.8 FlashAPI75.6–2None
7GPT-6 AstraChatGPT69.6–2None
8Gemini 3.5 Flash-LiteGemini49.4–2None

What you get for the cost

Haiku 5.5 · API scores within a point of Sonnet 5.5 · API at about a thirtieth of the cost.

Task score against relative cost for each setup. The table below has the same figures.4050607080901001×3×10×30×100×300×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreSonnet 5.5 · API: task score 93, 351× the cheapestHaiku 5.5 · API: task score 92, 11× the cheapestGPT-6.1 Sol · API: task score 89, 122× the cheapestGPT-6 Luna · API: task score 85, 7.1× the cheapestOpus 5.5 · Claude: task score 77, 65× the cheapest (2 of 2 outputs estimated)Gemini 3.8 Flash · API: task score 76, 74× the cheapestGPT-6 Astra · ChatGPT: task score 70, 19× the cheapest (2 of 2 outputs estimated)Gemini 3.5 Flash-Lite · Gemini: task score 49, the cheapest (2 of 2 outputs estimated)Sonnet 5.5 · APIHaiku 5.5 · APIGPT-6.1 Sol · APIGPT-6 Luna · APIOpus 5.5 · ClaudeGemini 3.8 Flash · APIGPT-6 Astra · ChatGPTGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Getting a clickable prototype in front of users or stakeholders fast.

Why it matters

A prototype that looks right but whose core interaction is broken wastes a research session. This task rewards working behaviour over polish.

What good looks like

  • The core interaction works end to end
  • Respects the stated constraints and scope
  • Handles the empty and error states named in the brief
  • Says what it's testing and what's faked
  • Looks like something you'd share: clear hierarchy, none of the stock AI-template look

Deliberately not measured

  • Production code quality
  • Design to a requested style: the brief never asks for one, so this measures default taste
Capability tested

Turning a spec into a usable interactive artefact

The failure we’re looking for

An attractive shell with a broken or absent core interaction

Grading

Browser checks (interaction and design) and decision model, calibrated against a blind PM review