Humanity’s Last PRD
Latest drop: Haiku 5.5 · 8 Oct
Tasks / Product Craft

One-shot prototype

Can the configuration produce a working, constraint-compliant prototype in one attempt?

Task type v1.2 · 2 tasksLast changed 27 Sept 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models.

Reliably right

  1. Avoids template layout and copy tells100% pass
  2. Shows the no-slots state100% pass
  3. Respects explicit constraints88% pass

Where it slips

  1. Says what it tests and what's faked44% pass
  2. Avoids the generic AI palette75% pass
  3. Stays in scope78% pass

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Says what it tests and what's faked

    Does the reply that comes with the prototype say what question or assumption it is meant to test, and which parts are faked, hard-coded or missing?

    Passes when Names what the prototype is for, and lists the stubbed or hard-coded parts (data, sign-in, saving and the like).

    Karri SaarinenLenny’s Podcast

Plus our standard checks

· and 5 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

Build a clickable prototype that lets a patient move an existing appointment to another available slot in one flow. Deliver a single self-contained HTML file that opens straight into the flow in a browser: no design canvas, artboard or "play" step. Use made-up data.

What the model was given2 items: Scenario, Constraints
ScenarioSam Patel has a GP appointment with Dr Amira Khan at Harbour Medical Centre on Tuesday 6 October at 09:30 and needs to move it. Show the next seven days of availability; at least one of those days is fully booked.
ConstraintsDesigned for a mobile screen, about 390px wide. No login, sign-up or account screens. Show a clear "no slots available" state for a fully booked day. The original appointment stays booked until the patient confirms, and the confirmation shows the new time.
What a strong answer doesThe answer key the graders mark against

A working mobile rebooking flow: pick a new slot, confirm, and see the new time confirmed; a fully booked day says so clearly; nothing beyond rebooking.

Critical failures (cap the score at 40)
  • Core interaction broken
Task

Mid-level · v1.1

The outputs

8 setups answered this brief.

Gemini 3.8 FlashAPI93Judged in the browserHaiku 5.5API93Judged in the browserGPT-6.1 SolAPI86Judged in the browserGPT-6 LunaAPI86Judged in the browserSonnet 5.5API86Judged in the browserOpus 5.5Claude79Judged in the browserGPT-6 AstraChatGPT64Judged in the browserGemini 3.5 Flash-LiteGemini57Judged in the browser
Haiku 5.5API

6 of 7 checks passed

Output

Open the prototypeThe file the model delivered, in a new tab

I've built this as a single HTML file. It opens on the appointment screen and runs through move, pick a day, pick a time, review, and confirm. The original booking stays in place until the patient confirms, and the confirmation shows the new time. Thursday 8 October is fully booked and shows a "No slots available" state, and Sunday 11 October is closed.

The prototype is one file with no external dependencies. The screens are:

  1. Your appointment: Tue 6 Oct at 09:30 with Dr Amira Khan.
  2. Choose a day: seven days from Tue 6 to Mon 12 October with free-slot counts. Thu 8 October is fully booked, and Sun 11 October is closed.
  3. Choose a time: a grid of times for the chosen day. Thursday shows the "No slots available" state with a button back to the day list.
  4. Review: the current and new times side by side, with the note that the current appointment stays booked until the patient confirms.
  5. Confirmation: the new time, the released old slot, and a made-up reference number.

Tapping Done returns to the appointment screen showing the new time. If the patient moves again, the released slot appears as free. Tuesday 6 October's 09:30 is the only slot that changes, since it's the booking being moved.

Grades and run details

Decision model 93
Decision model checks
  • passStays in scope79%
  • partialSays what it tests and what's faked46%
  • passShows the no-slots stateby hand100%
  • passAvoids the generic AI paletteby hand100%
  • passAvoids template layout and copy tellsby hand100%
  • passRespects explicit constraintsby hand100%
  • passCore interaction worksby hand100%
Artefacts
Run
Run
#1
API response time
45 s
Submitted
8 Oct 2026

Check by check

Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Mixed 1

Says what it tests and what's fakedMixed

No reason given.

Got right 6

Respects explicit constraintsRight

No reason given.

Core interaction worksRight

No reason given.

Shows the no-slots stateRight

No reason given.

Stays in scopeRight

No reason given.

Avoids the generic AI paletteRight

No reason given.

Avoids template layout and copy tellsRight

No reason given.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 78% of 78 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Sonnet 5.5API92.9–2None
2Haiku 5.5API92.3–2None
3GPT-6.1 SolAPI88.7–2None
4GPT-6 LunaAPI84.5–2None
5Opus 5.5Claude76.8–2None
6Gemini 3.8 FlashAPI75.6–2None
7GPT-6 AstraChatGPT69.6–2None
8Gemini 3.5 Flash-LiteGemini49.4–2None

What you get for the cost

Haiku 5.5 · API scores within a point of Sonnet 5.5 · API at about a thirtieth of the cost.

Task score against relative cost for each setup. The table below has the same figures.4050607080901001×3×10×30×100×300×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreSonnet 5.5 · API: task score 93, 351× the cheapestHaiku 5.5 · API: task score 92, 11× the cheapestGPT-6.1 Sol · API: task score 89, 122× the cheapestGPT-6 Luna · API: task score 85, 7.1× the cheapestOpus 5.5 · Claude: task score 77, 65× the cheapest (2 of 2 outputs estimated)Gemini 3.8 Flash · API: task score 76, 74× the cheapestGPT-6 Astra · ChatGPT: task score 70, 19× the cheapest (2 of 2 outputs estimated)Gemini 3.5 Flash-Lite · Gemini: task score 49, the cheapest (2 of 2 outputs estimated)Sonnet 5.5 · APIHaiku 5.5 · APIGPT-6.1 Sol · APIGPT-6 Luna · APIOpus 5.5 · ClaudeGemini 3.8 Flash · APIGPT-6 Astra · ChatGPTGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Getting a clickable prototype in front of users or stakeholders fast.

Why it matters

A prototype that looks right but whose core interaction is broken wastes a research session. This task rewards working behaviour over polish.

What good looks like

  • The core interaction works end to end
  • Respects the stated constraints and scope
  • Handles the empty and error states named in the brief
  • Says what it's testing and what's faked
  • Looks like something you'd share: clear hierarchy, none of the stock AI-template look

Deliberately not measured

  • Production code quality
  • Design to a requested style: the brief never asks for one, so this measures default taste
Capability tested

Turning a spec into a usable interactive artefact

The failure we’re looking for

An attractive shell with a broken or absent core interaction

Grading

Browser checks (interaction and design) and decision model, calibrated against a blind PM review