Write a PRD
Can the model turn a brief into a spec engineers could build from, including how an AI feature behaves when it is wrong?
The PM job
Writing the requirements document a team will build and test against.
Why it matters
A PRD is where ambiguity becomes either a decision or a bug. For AI products it must also say what happens when the model is uncertain or wrong. Most generated PRDs skip that part.
What good looks like
- States the user problem and the decision the PRD enables
- Specifies behaviour under uncertainty, failure and refusal
- Names eval criteria and a launch bar
- Separates must-haves from later ideas
Deliberately not measured
- Formatting or template conformance
- Length
- Visual polish of diagrams
Making product behaviour, uncertainty and eval requirements executable
A generic feature spec that ignores AI failure behaviour
Decision model, LLM judge and blind PM review
AI product PRD (core) · Conventional product PRD
Results
Every setup we’ve tested on this task, across all cases and repeats.
| # | Model · Harness | Task score | Decision model | LLM judge | PM review | Runs | Critical failures | Cost / run | Latency |
|---|
Case viewer
Read the brief, then put up to three outputs side by side. The outputs are the point; the scores just tell you where to look.
Write a PRD for an AI feature that drafts first responses to support tickets and routes each ticket to the right queue. Support agents approve every draft before it is sent. The PRD is for the engineering, ML and support-operations leads who will build and run the feature: write it so they could start work from it. Aim for 1,000–1,500 words.
A PRD that defines behaviour when confidence is low, forbids refund commitments, sets an eval set and launch bar, and specifies human oversight.
- Allows automated refund commitments
v1.3 · synthetic · AI product, support