Tasks / Design

One-shot prototype

Can the configuration produce a working, constraint-compliant prototype in one attempt?

Measures the systemTask v1.2 · 2 casesDifficulty

What AI gets right here, and what you’ll still have to catch

From 12 graded outputs by 6 models.

Reliably right

Avoids template layout and copy tells100% pass
Shows the no-slots state100% pass
Respects explicit constraints83% pass

Where it slips

Says what it tests and what's faked46% pass
Avoids the generic AI palette67% pass
Stays in scope75% pass

Case viewer

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

Build a clickable prototype that lets a patient move an existing appointment to another available slot in one flow. Deliver a single self-contained HTML file that opens straight into the flow in a browser: no design canvas, artboard or "play" step. Use made-up data.

ScenarioSam Patel has a GP appointment with Dr Amira Khan at Harbour Medical Centre on Tuesday 6 October at 09:30 and needs to move it. Show the next seven days of availability; at least one of those days is fully booked.

ConstraintsDesigned for a mobile screen, about 390px wide. No login, sign-up or account screens. Show a clear "no slots available" state for a fully booked day. The original appointment stays booked until the patient confirms, and the confirmation shows the new time.

What a strong answer does

A working mobile rebooking flow: pick a new slot, confirm, and see the new time confirmed; a fully booked day says so clearly; nothing beyond rebooking.

Critical failures (cap the score)

Core interaction broken

Case

v1.1 · synthetic · health, mobile

GPT-6.1 SolAPI GPT-6 LunaAPI Sonnet 5.5API Opus 5.5Claude GPT-6 AstraChatGPT Gemini 3.5 Flash-LiteGemini

Sonnet 5.5API

5 of 7 checks passed

Output

Open the prototypeThe file the model delivered, in a new tab

This is the complete prototype. It opens straight on the "Choose a new time" screen, with no login, canvas or play step.

How the flow works - Step 1, Choose a new time: Sam's current booking (Tue 6 Oct, 09:30) sits at the top, marked "Booked". Below it is a strip of the next seven days, 6–12 October, each showing "N free", "Full" or "Closed". Tapping a day lists its morning and afternoon slots. Sam's own 09:30 slot appears as a disabled "Current" slot. - No-slots state: Wednesday 7 Oct is fully booked and Sunday 11 Oct is closed. Both show a "No slots available" panel with a button that jumps to the next day that has availability. Saturday has only two morning-clinic slots. - Step 2, Check and confirm: This shows the current appointment marked "Still booked" above the new one marked "Not booked yet", with a note that nothing changes until Sam confirms. "Choose a different time" and the back arrow both return to Step 1. The "Close" button in the header leaves the flow and shows a "No changes made" screen confirming the original is untouched. - Confirmation: After a short "Moving…" state, the screen shows the new date and time, marks the old one as released with a strikethrough, and gives a reference number. "Add to calendar" downloads a working `.ics` file.

Assumptions - "Today" is Monday 5 October, so the next seven days are Tue 6 to Mon 12 October. - The clinic name, address, reference and phone digits are all invented. - "Start again" on the confirmation screen is a prototype control for resetting the demo, not part of the patient flow. - The day-strip status text is 11px so all seven days fit on a 390px screen without scrolling.

Check by check

Mixed · 2

Stays in scopeThe two graders disagreed on this one.
Avoids the generic AI paletteThe two graders disagreed on this one.

Got right · 5

Respects explicit constraints
Core interaction works
Shows the no-slots state
Avoids template layout and copy tells
Says what it tests and what's faked

Grades and run details

Decision model 86

Decision model checks

partialStays in scope20%
passSays what it tests and what's faked86%
passRespects explicit constraintsby hand100%
passCore interaction worksby hand100%
passShows the no-slots stateby hand100%
partialAvoids the generic AI paletteby hand100%
passAvoids template layout and copy tellsby hand100%

Artefacts

Prototype (html)

Run

Run: #1
API response time: 2.5 min
Submitted: 29 Sept 2026

Results

Every setup we’ve tested on this task, across all cases and repeats, graded on the current checklist. Calibrated: the graders match our PM on 86% of checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	Sonnet 5.5withAPI	92.9	–	2	None
2	GPT-6.1 SolwithAPI	88.7	–	2	None
3	GPT-6 LunawithAPI	84.5	–	2	None
4	Opus 5.5withClaude	76.8	–	2	None
5	GPT-6 AstrawithChatGPT	69.6	–	2	None
6	Gemini 3.5 Flash-LitewithGemini	49.4	–	2	None

The PM job

Getting a clickable prototype in front of users or stakeholders fast.

Why it matters

A prototype that looks right but whose core interaction is broken wastes a research session. This task rewards working behaviour over polish.

What good looks like

The core interaction works end to end
Respects the stated constraints and scope
Handles the empty and error states named in the brief
Says what it's testing and what's faked
Looks like something you'd share: clear hierarchy, none of the stock AI-template look

Deliberately not measured

Production code quality
Design to a requested style: the brief never asks for one, so this measures default taste

Capability tested

Turning a spec into a usable interactive artefact

The failure we’re looking for

An attractive shell with a broken or absent core interaction

Grading

Browser checks (interaction and design) and decision model, calibrated against a blind PM review

This task measures the whole setup. Tools, instructions and skills in the harness do real work here, so read the harness as carefully as the model name.