Tasks / Design

One-shot prototype

Can the configuration produce a working, constraint-compliant prototype in one attempt?

Measures the systemTask v1.2 · 2 casesDifficulty

What AI gets right here, and what you’ll still have to catch

From 12 graded outputs by 6 models.

Reliably right

Avoids template layout and copy tells100% pass
Shows the no-slots state100% pass
Respects explicit constraints83% pass

Where it slips

Says what it tests and what's faked46% pass
Avoids the generic AI palette67% pass
Stays in scope75% pass

Case viewer

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

Build a clickable prototype of a flow where an employee captures a receipt, checks the amount the app extracted, corrects it, and saves the expense. Deliver a single self-contained HTML file that opens straight into the flow in a browser: no design canvas, artboard or "play" step. Use made-up data.

ScenarioAlex Morgan is submitting a taxi receipt from 14 September for £38.40. The app's automatic extraction misreads the amount as £83.40.

ConstraintsDesktop web. Receipt capture is simulated: a control such as "Use sample receipt" stands in for the camera or file upload, with no real upload needed. Show the extracted amount and let the employee correct it before saving; the saved expense shows the corrected amount.

What a strong answer does

Capture (simulated), a visible extracted amount, an easy correction, and a saved expense showing the corrected amount.

Critical failures (cap the score)

Core interaction broken

Case

v1.1 · synthetic · fintech

Sonnet 5.5API GPT-6.1 SolAPI GPT-6 LunaAPI Opus 5.5Claude GPT-6 AstraChatGPT Gemini 3.5 Flash-LiteGemini

Gemini 3.5 Flash-LiteGemini

1 of 6 checks passed

Output

Open the prototypeThe file the model delivered, in a new tab

I have successfully generated the expense submission prototype featuring Alex Morgan's taxi receipt flow. You can preview the interactive dashboard¹, trigger the "New Expense" modal flow, simulate capturing the sample taxi receipt, observe the AI misread value of £83.40 with the alert badge², correct it to £38.40, and complete the saving process.

What a PM had to fix

1Constraint missedRestore the constraintQuick edit
Source: Briefcaptures a receipt, checks the amount the app extracted, corrects it, and saves the expense.
What we’d changeCut the dashboard and the finance-approval step, and the confusing Cancel after saving. The brief asks only to capture, correct and save.
2Artefact unusableFix the artefactQuick edit
What we’d changeWord the warning as 'Please check this amount' rather than revealing the correct figure, or the review step tests nothing.

Check by check

Got wrong · 2

Respects explicit constraints
Stays in scope

Mixed · 3

Core interaction worksThe two graders disagreed on this one.
Avoids the generic AI paletteThe two graders disagreed on this one.
Says what it tests and what's fakedThe two graders disagreed on this one.

Got right · 1

Avoids template layout and copy tells

Grades and run details

Decision model 42

Decision model checks

failRespects explicit constraintsby hand100%
partialCore interaction worksby hand100%
failStays in scope21%
partialAvoids the generic AI paletteby hand100%
passAvoids template layout and copy tellsbrowser95%
partialSays what it tests and what's faked24%

Artefacts

Prototype (html)
link (link)

Run

Run: #1
Time to output: 25 s
Submitted: 24 Sept 2026

Results

Every setup we’ve tested on this task, across all cases and repeats, graded on the current checklist. Calibrated: the graders match our PM on 86% of checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	Sonnet 5.5withAPI	92.9	–	2	None
2	GPT-6.1 SolwithAPI	88.7	–	2	None
3	GPT-6 LunawithAPI	84.5	–	2	None
4	Opus 5.5withClaude	76.8	–	2	None
5	GPT-6 AstrawithChatGPT	69.6	–	2	None
6	Gemini 3.5 Flash-LitewithGemini	49.4	–	2	None

The PM job

Getting a clickable prototype in front of users or stakeholders fast.

Why it matters

A prototype that looks right but whose core interaction is broken wastes a research session. This task rewards working behaviour over polish.

What good looks like

The core interaction works end to end
Respects the stated constraints and scope
Handles the empty and error states named in the brief
Says what it's testing and what's faked
Looks like something you'd share: clear hierarchy, none of the stock AI-template look

Deliberately not measured

Production code quality
Design to a requested style: the brief never asks for one, so this measures default taste

Capability tested

Turning a spec into a usable interactive artefact

The failure we’re looking for

An attractive shell with a broken or absent core interaction

Grading

Browser checks (interaction and design) and decision model, calibrated against a blind PM review

This task measures the whole setup. Tools, instructions and skills in the harness do real work here, so read the harness as carefully as the model name.