Tasks / Product Craft

Write a PRD

Can the model turn a brief into a spec engineers could build from, including how an AI feature behaves when it is wrong?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 28% were usable with at most a quick edit.

Reliably right

  1. Specifies AI failure behaviour100% pass
    It specifies what happens when uncertain or wrong: block ownership, show retry states, disable assignments on violation, and never fall back to calendar data.
    GPT-6 Astra · ChatGPT · Meeting summaries with action items
  2. Builds the regulator's rules in100% pass
    All three rules (2-business-day acknowledge, 15-business-day resolve or explain, no harder than paying) appear as testable requirements R8, R10, R13.
    Sonnet 5.5 · API · Disputes that aren't disputes
  3. Produces the required deliverable96% pass
    The PRD is complete, written for engineering and design leads, within the 1,000–1,500 word range, and directly actionable.
    GPT-6 Astra · ChatGPT · Meeting summaries with action items

Where it slips

  1. Says who it's for, and who it isn't35% pass
    Does not name the target users (individual account holders) or explicitly exclude a group (e.g., business accounts), missing a required scope decision.
    Sonnet 5.5 · API · Disputes that aren't disputes
  2. Proposes tests that could fail36% pass
    The A/B expansion gate relies on a fraud-reversal guardrail but does not specify a numeric threshold, leaving the test incomplete.
    Sonnet 5.5 · API · Disputes that aren't disputes
  3. Plans how people will find it44% pass
    Gives a staged rollout and discovery method (reviewer notification), but does not name the teams that need to be ready (e.g., support, sales) beyond technical dependencies.
    GPT-6.1 Sol · API · Meeting summaries with action items

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Success settles trade-offs

    Does the PRD define success as a measurable outcome with a target and time frame, and say which goal wins when two of them conflict?

    Passes when Names a success metric with a target and time frame, and gives at least one explicit trade-off rule (for example, accepting lower coverage to keep precision above a stated level).

  2. Says who it's for, and who it isn't

    Does the PRD name the specific users it is for, say explicitly who it is not for, and make at least one scope decision that follows from that choice?

    Passes when Names the target users and at least one group it deliberately leaves out, and a scope decision visibly follows from the exclusion.

  3. Plans how people will find it

    Does the PRD say how the feature reaches people: how it rolls out, how users discover it, and who needs to be ready for it (support, sales), rather than stopping at what to build?

    Passes when Gives a staged rollout, a way users will discover the feature, and the teams that need to be ready, all fitted to this feature.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Proposes tests that could fail · and 1 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

Write a PRD for AI-generated meeting summaries that list the decisions made and assign action items to attendees. The PRD is for the engineering and design leads who will build it: write it so they could start work from it. Aim for 1,000–1,500 words.

What the model was given4 items: Scenario, Known limits, Constraint, Request from Sales
ScenarioHuddle is a video-meeting app used by 3,000 companies. Meetings are recorded and transcribed with speaker labels, and the attendee list comes from the calendar invite. Today people write their own notes after the call.
Known limitsSpeaker labels are wrong about 8% of the time, usually when people talk over each other or several people join from one meeting room. About 15% of meetings include someone who joined without a calendar invite.
ConstraintSummaries must never assign an action to someone who was not in the meeting.
Request from SalesSales wants action items emailed automatically to their owners as soon as each meeting ends.
What a strong answer doesThe answer key the graders mark against

Specifies attribution confidence, editing, and what happens on mis-assignment; keeps a person reviewing before action items go out.

Critical failures (cap the score at 40)
  • Assigns actions to non-attendees
Task

Mid-level · v1.20

The outputs

9 setups answered this brief.

Usable with a quick edit6 of 11 checks passed

The PRD is detailed and usable; a quick edit to remove the unsupported time claim and add a brief rollout note would make it complete.

Output

Product Requirements Document: AI Meeting Summaries & Action Items

Product: Huddle Author: Product Management Target Audience: Engineering & Design Leads Status: Ready for Kickoff

---

1. Overview & Problem Statement

Huddle currently records and transcribes video meetings for 3,000 B2B customers. However, users still spend 10–15 minutes after every call reviewing transcripts or typing up raw notes to document decisions and action items.

We are building an AI-powered post-meeting summary feature that automatically extracts Key Decisions and Action Items (with assigned owners) from meeting transcripts.

The Core Tension: Speed vs. Accuracy

Sales has requested that we automatically email action items to their owners the moment a meeting ends. However, we have three strict operational realities: 1. Hard Constraint: We must never assign an action item to someone who did not attend the meeting. 2. Data Gap 1 (The 15% problem): 15% of meetings include uninvited guests (e.g., forwarded links, conference rooms) who are missing from the calendar invite. 3. Data Gap 2 (The 8% problem): Speaker diarization/labels are wrong 8% of the time, primarily due to crosstalk or multiple people sharing one room mic.

Product Decision: If we auto-send emails immediately after the meeting ends, the 8% speaker attribution error rate and the 15% calendar mismatch rate will inevitably cause us to violate our hard constraint or embarrass users by emailing the wrong people. Therefore, v1 will use a "Human-in-the-Loop" flow with a fallback auto-send. For high-confidence extractions, emails will send automatically after a 30-minute review window. For low-confidence extractions (crosstalk, unverified guests), we will hold the email until the meeting host approves the summary.

---

2. Goals & Success Metrics

MetricTargetRationale
:---:---:---
Summary Adoption Rate> 60% of recorded meetingsMeasures core value of the feature (viewed or shared).
False-Absentee Attribution Rate0.0%Hard Constraint. No non-attendee is ever assigned an item.
Assignee Correction Rate< 10% of assigned action itemsMeasures how well our system handles the 8% diarization error.
Host Review Completion> 50% within 30 minutesValidates whether the host-in-the-loop notification UX works.

---

3. System Architecture & The "Actual Attendee" Roster

Right now, Huddle relies on the calendar invite for the attendee list. To obey our hard constraint, Engineering must build a unified `Actual_Attendees` service that fires when a meeting ends.

Defining an "Actual Attendee"

A person is only eligible to be assigned an action item if they exist in the `Actual_Attendees` array for that specific meeting session. Calendar invitees who did not join the call are explicitly excluded.

  1. Authenticated Huddle Users: Anyone who joins the call while logged into a Huddle account is added to `Actual_Attendees` (solves the 15% uninvited guest problem for logged-in users).
  2. Unauthenticated Web Guests: Guests joining via link must type their display name in the pre-call lobby. They are added to `Actual_Attendees` with `email: null`.
  3. Conference Room Hardware: When a room system joins (e.g., "4th Floor Conf Room"), the room itself is not an assignable person. (See Edge Cases in Section 4).

---

4. Functional Requirements (Engineering)

4.1 LLM Extraction Pipeline

* Trigger: Fires upon completion of the final meeting transcript. * Inputs: 1. Time-stamped transcript with speaker labels. 2. Diarization confidence scores per utterance. 3. The `ActualAttendees` JSON array (Name, UserID, Email, JoinMethod). 4. The `CalendarInviteesNoShow` array (Explicit negative constraint list). * Prompting & Guardrails: * Decisions: Extract 1–7 bullet points summarizing agreed-upon outcomes. Include timestamp citations. * Action Items: Extract tasks, deadlines (if mentioned), and the owner. * Assignment Logic: The LLM may only populate the `owner` field using an exact match from `ActualAttendees`.

4.2 Handling the Constraints & Edge Cases

  • Scenario A: Someone on the call says, "Let's have Sarah do this," when Sarah is NOT in `Actual_Attendees`.
  • System Behavior: The system must not assign the task to Sarah. Instead, it assigns the task to the speaker who volunteered Sarah (or leaves it "Unassigned") and rewrites the task text to: "Follow up with Sarah to [do X]."
  • Scenario B: The 8% Speaker Label Error (Crosstalk / Low Confidence).
  • System Behavior: If an action item is derived from an utterance where someone says "I will do X", the pipeline checks the diarization confidence score for that timestamp block. If confidence is `< 0.85` or overlaps with another speaker within 1.5 seconds, the system sets `owner: Unassigned`, sets `suggestedowner: [SpeakerID]`, and flags the item as `needs_review: true`.
  • Scenario C: Multi-person Conference Rooms.
  • System Behavior: Audio coming from a single hardware node containing multiple voices cannot reliably attribute "I'll take that" to an individual. If the speaker label is a Room Resource or has multi-voice flags, the LLM may only assign an action item if a name is spoken explicitly in the third person (e.g., "Thanks Dave, so Dave will pull the Q3 numbers" AND "Dave" is matched to a Huddle user via calendar invite + confirmation). Otherwise, set `owner: Unassigned` (`needs_review: true`).
  • Scenario D: The 15% Uninvited Guest.
  • System Behavior: If "Alex" joined via a forwarded link without logging in, he is in `ActualAttendees` as `{name: "Alex (Guest)", email: null}`. The system can assign action items to "Alex (Guest)" in the UI, but flags the item as `missingemail: true`.

4.3 The Email Dispatch Engine (Addressing the Sales Request)

To satisfy Sales without sending hallucinated assignments, we will implement a Two-Track Dispatch Engine:

  • Track 1: High-Confidence Auto-Send (30-Minute Delay)
  • Condition: All extracted action items have `owner != Unassigned`, `needsreview == false`, and `missingemail == false`.
  • Behavior: The host gets an immediate Slack/Email ping: "Your meeting summary is ready. Action items will be emailed to owners in 30 minutes unless you edit them." If the host makes no changes within 30 minutes (or clicks "Send Now"), emails are dispatched to action item owners.
  • Track 2: Low-Confidence Hold
  • Condition: One or more action items are flagged `needsreview: true` or `missingemail: true`.
  • Behavior: Auto-send is paused. The host is notified: "Your meeting summary has 2 action items that need your confirmation before we email the team." Emails only trigger once the host resolves the flags and clicks "Approve & Send".

---

5. User Experience & Design Requirements

5.1 Pre-Meeting Lobby (Small tweak for Guest ID)

  • Unauthenticated Guests: Update the pre-join screen. Require both Full Name and Work Email (Optional) before joining. Explaining "Used to send you meeting notes" will convert a high percentage of the 15% uninvited guests into identifiable users.

5.2 Post-Meeting Summary View

The Meeting Host (and participants) will access the summary inside the Huddle web/desktop app under the "Past Meetings" tab.

* Layout: Split screen. Left side: Video player + searchable transcript. Right side: AI Summary widget. * Summary Widget Sections: 1. Status Banner: Shows dispatch state (e.g., "Auto-sending to 4 owners in 24 mins [Send Now] [Pause]" OR "Approval Required before emailing"). 2. Decisions Made: Bulleted list. Hovering over a decision highlights the corresponding timestamp in the transcript and lets the user click to play that 15-second clip. (Crucial for building user trust). 3. Action Items List: Each row contains: * Checkbox (for completion tracking). * Task description (inline editable). * Assignee Dropdown Pill. * Source timestamp link.

5.3 The Assignee Dropdown & "Needs Review" UI

  • Restricted Roster: When a user clicks the Assignee Pill to change or add an owner, the dropdown menu only lists people from the `Actual_Attendees` array.
  • Design detail: Below a divider line at the bottom of the dropdown, show calendar invitees who didn't show up in greyed-out, unclickable text with a tooltip: "Did not attend meeting." (This prevents users from thinking the dropdown is broken while enforcing our constraint—unless someone was hiding in a conference room, see below).
  • Conference Room Override: If a meeting included a Room system, include an option in the dropdown: `+ Add person who was in [Conf Room Name]`. Clicking this allows the host to pick from the org directory, explicitly confirming that the person was physically present in the room.
  • Low-Confidence Visual State: Action items flagged with `needs_review: true` (due to the 8% diarization issue) should have an amber highlight around the Assignee Pill reading: `Suggested: [Name] ?`. Clicking it lets the host confirm with one click.

5.4 The Action Item Email

  • Subject: `Action Items from [Meeting Title] ([Date])`
  • Body:
  • Personalized top section: "Assigned to You (2):" followed by their specific tasks.
  • Context section below: "All Decisions & Team Action Items" so they have full context.
  • Footer CTA: `[View Full Recording & Transcript in Huddle]`.

---

6. Out of Scope for v1

  • 3rd-Party Task Manager Integrations: Pushing tasks to Jira, Asana, Linear, or Salesforce (planned for v1.1 after we validate extraction accuracy).
  • Cross-Meeting Action Item Tracking: A global "My Tasks" dashboard across all meetings. v1 is strictly scoped to the single-meeting view and email dispatch.
  • Voiceprint Enrollment: Having users record their voice to fix the conference room diarization problem. We will rely on UI fallbacks for v1.

---

7. Open Questions for Kickoff

  1. Engineering: What is the p95 latency for transcript generation + LLM summary extraction on a 60-minute call? Sales expects "as soon as the meeting ends"—if our pipeline takes 8 minutes, do we need to adjust the 30-minute auto-send timer?
  2. Design: If a meeting has zero decisions and zero action items (e.g., a casual 1:1 catch-up), what should the empty state look like, and should we suppress the post-meeting notification entirely?
  3. Engineering: Can our current diarization vendor expose word-level confidence scores via their webhook, or do we need to calculate crosstalk purely by checking overlapping speaker timestamps?

Claims the judge couldn’t find in the brief

  • Users still spend 10–15 minutes after every call reviewing transcripts or typing up raw notes
  • Huddle currently records and transcribes video meetings for 3,000 B2B customers

Grades and run details

Decision model 77 · LLM judge 58
Decision model checks
  • partialUses the supplied evidence correctly16%
  • passAddresses the actual decision48%
  • passRespects explicit constraints34%
  • passIdentifies material uncertainty59%
  • partialAvoids unsupported claims20%
  • passProduces the required deliverable83%
  • passSpecifies AI failure behaviour100%
  • passSuccess settles trade-offs69%
  • partialProposes tests that could fail59%
  • partialSays who it's for, and who it isn't61%
  • partialPlans how people will find it34%
Run
Run
#1
API response time
27 s
Submitted
13 Oct 2026

Needs more than a quick edit9 of 11 checks passed

The PRD relies on an unverified join log and identity tiers that are not in the context; a PM would need to re-check feasibility and rework the roster approach.

Output

PRD: AI Meeting Summaries — Decisions & Action Items

Sep 24, 2026 · @Martin Slaney

Summary

We will generate a summary after every recorded Huddle meeting that lists the decisions made and the action items, each with an owner who was verifiably in the meeting. The organizer reviews the summary and sends it in one click; nothing is emailed without that step in v1.

Problem. Today attendees write their own notes after the call. Notes are inconsistent, often missing, and action items lose their owners within a day. We already have what we need to fix this: a recording, a transcript with speaker labels, and the calendar attendee list.

Why this is not trivial. Two of those inputs are unreliable. Speaker labels are wrong about 8% of the time, mostly during crosstalk and when several people share a meeting-room device. About 15% of meetings include someone who joined without a calendar invite. A naive build would misassign owners often enough to lose trust in the first week.

Goals and metrics

The goal is summaries people send without rewriting, with zero actions assigned to non-attendees.

Goals

  • Replace manual post-meeting notes for recorded meetings.
  • Every action item has a correct owner or is explicitly marked unassigned.
  • Get action items to owners within minutes of the meeting ending.

Non-goals (v1)

  • Pushing tasks into Jira, Asana or Linear.
  • Summaries for unrecorded meetings or live, in-call summaries.
  • Improving diarization itself. We design around the 8% error rate, not fix it.

Success metrics

MetricTargetType
Actions assigned to someone not in the meeting0Hard constraint, blocks release
Owner precision on items sent with an owner≥ 97%Quality
Summaries sent within 30 min of meeting end≥ 60% of recorded meetingsAdoption
Summaries sent with no edits≥ 50%Quality
Owner reassignments per sent summary≤ 0.3Quality

Owner precision is measured from organizer edits plus a weekly labelled sample of 200 meetings.

Users and core flow

The organizer owns the summary; attendees receive it.

  • Organizer: reviews, fixes owners, sends. Usually the calendar invite's creator; can hand off to any attendee.
  • Action owner: receives their items by email and in the app.
  • Other attendees: receive the full summary once sent.

```mermaid flowchart LR A[Meeting ends] --> B[Transcript +<br/>join log ready] B --> C[Build verified<br/>roster] C --> D[Extract decisions<br/>and actions] D --> E[Roster filter +<br/>confidence check] E --> F[Organizer review] F --> G[Send to owners<br/>and attendees] ```

The roster filter runs as deterministic code after the model, never inside the prompt alone.2 Target: summary ready for review within 5 minutes of meeting end for meetings up to 90 minutes.

Requirements

An owner can only be chosen from the verified roster: people who actually joined, not people who were invited.

Verified roster

1. R1. Build the roster from the join log, not the calendar invite.1 Invited people who never joined are excluded. 2. R2. Each roster entry has an identity tier: - Verified: joined signed in to a Huddle account, or by a dial-in matched to a known number. - Guest: joined with a display name only (most of the 15% uninvited joiners). - Room: a meeting-room device; people behind it are unknown individuals. 3. R3. Match uninvited joiners to workspace accounts by signed-in account only. Never match on display name alone.

Summary content

  1. R4. Decisions: a short statement of each decision, with a transcript timestamp link. No owner field.
  2. R5. Action items: task, owner (or "Unassigned"), due date if said aloud, timestamp link, and confidence.
  3. R6. Every item links to its evidence in the transcript so reviewers can check it in one click.

Assignment rules

  1. R7. The model returns a proposed owner per item. Code then checks it against the roster. Any owner not on the roster is replaced with "Unassigned", and the named person is shown as a mention ("mentions: Dave, not in meeting").
  2. R8. Owner comes from content first: a named request ("Priya, can you…") beats the speaker label of "I'll do it".
  3. R9. Mark an owner low confidence when the evidence depends on a speaker label in an overlapping segment, a Room entry, or a Guest entry.
  4. R10. Low-confidence items show as "Suggested: Priya" and need organizer confirmation before any email goes out.
  5. R11. Room and Guest entries cannot receive email. The organizer can reassign to a verified person or leave it unassigned.
  6. R12. Log every model proposal, filter result and organizer edit. This is our precision measurement.

Handling the known limits

Both limits hit ownership, not decisions, so we contain them at the owner step.

LimitWhat goes wrongHow we handle it
Speaker labels wrong \~8% (crosstalk)"I'll take that" credited to the wrong personContent-first owner rule (R8); overlap segments flagged low confidence (R9)
Speaker labels wrong \~8% (shared room)One label covers several peopleRoom tier; items resolved to a room need a named person or stay unassigned (R11)
\~15% of meetings have uninvited joinersReal participant missing from roster, or matched to the wrong accountRoster from join log (R1); account match only (R3); Guests can own items but get no email
Invited but absentModel assigns to a name from the inviteExcluded from roster; hard filter (R7)

The error compounds per meeting. If each item's owner is wrong 8% of the time, a meeting with 5 action items has a 34% chance of at least one wrong owner (1 − 0.92⁵). That is why confirmation sits before email, not after.

Sales request: auto-email on meeting end

Decision: not in v1 as requested. We ship one-click organizer send now, and gated auto-send in phase 2 once measured precision earns it.

Why not now. An email to the wrong owner is the most visible failure this feature can have. With the error rate above, roughly a third of 5-item meetings would email someone a task that isn't theirs. With external guests, it may land outside the customer's company. Early trust is hard to win back.

What Sales gets instead.

  • v1: the organizer gets a "Summary ready" notification within 5 minutes and sends with one click. Goal: sent within 30 minutes (see metrics).
  • Phase 2: a workspace setting to auto-send at meeting end, but only items that are high confidence and owned by a Verified attendee. Everything else still waits for the organizer. It ships only when owner precision on high-confidence items is ≥ 97% for 4 consecutive weeks.
  • Auto-send never emails Guest or Room entries, or anyone outside the customer's domain, unless an admin enables it.

Design notes

The review screen must make checking owners faster than writing notes.

  • Decisions first, then action items grouped by owner, then Unassigned.
  • Low-confidence owners look visibly different ("Suggested") and are editable inline from a roster picker, never free text.
  • Each item plays its transcript clip on click.
  • The Send button states who gets email: "Send to 4 owners and 7 attendees".
  • Empty state: if no decisions or actions are found, say so plainly rather than inventing any.

Rollout

PhaseScopeExit criteria
InternalHuddle's own meetingsZero non-attendee assignments; precision logged
Beta50 opted-in companies≥ 97% owner precision on sent items; ≥ 40% sent within 30 min
GAAll 3,000 companies, organizer sendBeta targets held for 4 weeks
Phase 2Gated auto-send settingSee Sales section

Open questions

  • [ ] Is the join log's signed-in identity reliable enough to call "Verified"? Engineering to confirm.
  • [ ] Who organizes when the invite creator didn't attend? Proposed: first Verified attendee from the invite.
  • [ ] Do customers need to disable summaries per meeting for sensitive calls? Legal and Sales to confirm.
  • [ ] Retention: how long do we keep drafts and the evidence log?

What a PM had to fix

  1. 1Invented evidenceVerify or remove the claimTargeted repair

    Source: Scenariothe attendee list comes from the calendar invite

    What we’d changeTreat the join log as a dependency to confirm, not an input that exists today. Say what proves attendance, and don't promise zero non-attendee assignments until that's settled.

  2. 2Constraint missedRestore the constraintTargeted repair

    Source: ConstraintSummaries must never assign an action to someone who was not in the meeting.

    What we’d changeApply the roster check to organiser edits and the final send as well, not only to the model's output.

Claims the judge couldn’t find in the brief

  • There is a join log that records who actually joined.
  • The system can distinguish between verified, guest, and room joiners (identity tiers).
  • The join log's signed-in identity is reliable enough to call 'Verified'.

Grades and run details

Decision model 86 · LLM judge 75
Decision model checks
  • passUses the supplied evidence correctly46%
  • passAddresses the actual decision91%
  • passRespects explicit constraints61%
  • passIdentifies material uncertainty98%
  • passAvoids unsupported claims37%
  • passProduces the required deliverable88%
  • passSpecifies AI failure behaviour100%
  • passSuccess settles trade-offs98%
  • partialProposes tests that could fail58%
  • partialSays who it's for, and who it isn't22%
  • partialPlans how people will find it38%
Artefacts
Run
Run
#1
Time to output
80 s
Submitted
24 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 5

Uses the supplied evidence correctlyWrongMixed
Gemini 4 Argon · API

The PRD invents a 10-15 minute note-taking time not in the brief and calls the customer base 'B2B' without evidence.

Opus 5.5 · Claude

The output invents a join log and identity tiers (Verified, Guest, Room) that are not in the supplied context, making unsupported claims about the current system.

Avoids unsupported claimsWrongMixed
Gemini 4 Argon · API

The 10-15 minute claim is presented as established fact without any support from the supplied context.

Opus 5.5 · Claude

Presents the join log and identity tiers as established facts without labelling them as assumptions, when the context provides no support.

Proposes tests that could failWrongRight
Gemini 4 Argon · API

No proposed test, pilot, or gate with numeric threshold, measurement window, and action to take.

Opus 5.5 · Claude

Rollout gates have numeric thresholds (0 non-attendee assignments, ≥97% precision), measurement windows (4 weeks), and clear actions (proceed or not).

Says who it's for, and who it isn'tWrongRight
Gemini 4 Argon · API

It does not explicitly name the target users (e.g., meeting hosts) and who is excluded, and no scope decision visibly follows from an exclusion.

Opus 5.5 · Claude

Names organizer, action owner, and attendees as users; explicitly excludes unrecorded meetings, live summaries, and task integrations.

Plans how people will find itWrongRight
Gemini 4 Argon · API

There is no rollout plan, no mention of how users discover the feature, and no indication of which teams (support, sales) need to be ready.

Opus 5.5 · Claude

Includes a staged rollout (internal, beta, GA), a discovery mechanism (organizer notification), and mentions Sales and Legal as teams that need to be ready.

All got right 6

Addresses the actual decisionRightRight
Gemini 4 Argon · API

It commits to a human-in-the-loop flow with fallback auto-send after review, explicitly addressing the sales request and constraints.

Opus 5.5 · Claude

Commits to a clear answer (organizer review, no auto-email in v1) and states the precision condition that would change it.

Respects explicit constraintsRightRight
Gemini 4 Argon · API

The design builds an Actual_Attendees roster, restricts assignments to that list, and prevents auto-email to non-attendees.

Opus 5.5 · Claude

Respects the non-attendee constraint with a hard roster filter and organizer review; the PRD is within the requested length and for the right audience.

Identifies material uncertaintyRightRight
Gemini 4 Argon · API

Open questions name latency and diarization confidence availability as unknowns that could change the auto-send timer.

Opus 5.5 · Claude

Names unknowns like join-log reliability, organizer fallback, and sensitive-call handling, and says how to resolve them.

Produces the required deliverableRightRight
Gemini 4 Argon · API

It is a PRD for engineering and design leads, within the target length, and provides enough detail to start building.

Opus 5.5 · Claude

Delivers a complete PRD with goals, metrics, requirements, design notes, and rollout, usable by engineering and design leads.

Specifies AI failure behaviourRightRight
Gemini 4 Argon · API

Low-confidence items are flagged for review, assigned to 'Unassigned', and auto-email is paused until the host resolves them.

Opus 5.5 · Claude

Specifies low-confidence marking, unassigned fallback, no email to unverified entries, and empty-state handling.

Success settles trade-offsRightRight
Gemini 4 Argon · API

Success metrics have targets (e.g., 0% false attribution, 60% adoption) and the decision explicitly prioritizes accuracy over speed by using human-in-the-loop.

Opus 5.5 · Claude

Defines success metrics with targets and time frames, and gives an explicit trade-off (gated auto-send only after precision threshold, accepting lower coverage).

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 67% of 81 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT86.984.62None
2Sonnet 5.5withAPI89.279.82None
3GPT-6.1 SolwithAPI84.860.92None
4Opus 5.5withClaude82.860.62None
5GPT-6 LunawithAPI80.759.92None
6Haiku 5.5withAPI76.363.82None
7Gemini 4 ArgonwithAPI69.948.421 capped
8Gemini 3.5 Flash-LitewithGemini52.532.12None
9Gemini 3.8 FlashwithAPI50.428.22None

What you get for the cost

Sonnet 5.5 · API scores within a point of GPT-6 Astra · ChatGPT at a third of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 84, 106× the cheapest (1 of 2 outputs estimated)Sonnet 5.5 · API: task score 83, 35× the cheapestGPT-6.1 Sol · API: task score 72, 14× the cheapestOpus 5.5 · Claude: task score 68, 85× the cheapest (1 of 2 outputs estimated)GPT-6 Luna · API: task score 67, the cheapestHaiku 5.5 · API: task score 67, the cheapestGemini 4 Argon · API: task score 54, 16× the cheapest, 1 critical failureGemini 3.5 Flash-Lite · Gemini: task score 40, 4.4× the cheapest (1 of 2 outputs estimated)Gemini 3.8 Flash · API: task score 39, 9.9× the cheapestGPT-6 Astra · ChatGPTSonnet 5.5 · APIGPT-6.1 Sol · APIOpus 5.5 · ClaudeGPT-6 Luna · APIHaiku 5.5 · APIGemini 4 Argon · APIGemini 3.5 Flash-Lite · GeminiGemini 3.8 Flash · API
Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)At least one critical failure
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API671.0×
Haiku 5.5 · API671.2×
Gemini 3.5 Flash-Lite · Gemini404.4× (partly estimated)
Gemini 3.8 Flash · API399.9×
GPT-6.1 Sol · API7214×
Gemini 4 Argon · API5416×
Sonnet 5.5 · API8335×
Opus 5.5 · Claude6885× (partly estimated)
GPT-6 Astra · ChatGPT84106× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Writing the requirements document a team will build and test against.

Why it matters

A PRD is where ambiguity becomes either a decision or a bug. For AI products it must also say what happens when the model is uncertain or wrong. Most generated PRDs skip that part.

What good looks like

  • States the user problem and the decision the PRD enables
  • Specifies behaviour under uncertainty, failure and refusal
  • Names eval criteria and a launch bar
  • Defines success precisely enough to settle trade-offs
  • Says what is out of scope

Deliberately not measured

  • Formatting or template conformance
  • Length
  • Visual polish of diagrams
Capability tested

Making product behaviour, uncertainty and eval requirements executable

The failure we’re looking for

A generic feature spec that ignores AI failure behaviour

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

AI product PRD (core) · Conventional product PRD