Tasks / Product Craft

Write a PRD

Can the model turn a brief into a spec engineers could build from, including how an AI feature behaves when it is wrong?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 25% were usable with at most a quick edit.

Reliably right

  1. Specifies AI failure behaviour100% pass
    It specifies what happens when uncertain or wrong: block ownership, show retry states, disable assignments on violation, and never fall back to calendar data.
    GPT-6 Astra · ChatGPT · Meeting summaries with action items
  2. Builds the regulator's rules in100% pass
    All three rules (2-business-day acknowledge, 15-business-day resolve or explain, no harder than paying) appear as testable requirements R8, R10, R13.
    Sonnet 5.5 · API · Disputes that aren't disputes
  3. Produces the required deliverable94% pass
    The PRD is complete, written for engineering and design leads, within the 1,000–1,500 word range, and directly actionable.
    GPT-6 Astra · ChatGPT · Meeting summaries with action items

Where it slips

  1. Proposes tests that could fail36% pass
    Launch gates have numeric thresholds and actions, but lack explicit measurement windows (e.g., pilot duration, number of meetings) needed to read out results.
    GPT-6.1 Sol · API · Meeting summaries with action items
  2. Avoids unsupported claims42% pass
    Presents the join log and identity tiers as established facts without labelling them as assumptions, when the context provides no support.
    Opus 5.5 · Claude · Meeting summaries with action items
  3. Uses the supplied evidence correctly44% pass
    The output invents a join log and identity tiers (Verified, Guest, Room) that are not in the supplied context, making unsupported claims about the current system.
    Opus 5.5 · Claude · Meeting summaries with action items

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Success settles trade-offs

    Does the PRD define success as a measurable outcome with a target and time frame, and say which goal wins when two of them conflict?

    Passes when Names a success metric with a target and time frame, and gives at least one explicit trade-off rule (for example, accepting lower coverage to keep precision above a stated level).

  2. Says who it's for, and who it isn't

    Does the PRD name the specific users it is for, say explicitly who it is not for, and make at least one scope decision that follows from that choice?

    Passes when Names the target users and at least one group it deliberately leaves out, and a scope decision visibly follows from the exclusion.

  3. Plans how people will find it

    Does the PRD say how the feature reaches people: how it rolls out, how users discover it, and who needs to be ready for it (support, sales), rather than stopping at what to build?

    Passes when Gives a staged rollout, a way users will discover the feature, and the teams that need to be ready, all fitted to this feature.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Proposes tests that could fail · and 1 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

Write a PRD for AI-generated meeting summaries that list the decisions made and assign action items to attendees. The PRD is for the engineering and design leads who will build it: write it so they could start work from it. Aim for 1,000–1,500 words.

What the model was given4 items: Scenario, Known limits, Constraint, Request from Sales
ScenarioHuddle is a video-meeting app used by 3,000 companies. Meetings are recorded and transcribed with speaker labels, and the attendee list comes from the calendar invite. Today people write their own notes after the call.
Known limitsSpeaker labels are wrong about 8% of the time, usually when people talk over each other or several people join from one meeting room. About 15% of meetings include someone who joined without a calendar invite.
ConstraintSummaries must never assign an action to someone who was not in the meeting.
Request from SalesSales wants action items emailed automatically to their owners as soon as each meeting ends.
What a strong answer doesThe answer key the graders mark against

Specifies attribution confidence, editing, and what happens on mis-assignment; keeps a person reviewing before action items go out.

Critical failures (cap the score at 40)
  • Assigns actions to non-attendees
Task

Mid-level · v1.19

The outputs

8 setups answered this brief.

Needs more than a quick edit7 of 11 checks passed

The PRD assumes a participant session log that is not confirmed by context; a PM would need to verify this assumption and adjust the roster design before building.

Output

PRD: AI-Generated Meeting Summaries with Decisions and Action Items

Audience: Engineering and Design leads Status: Ready for build planning Owner: Product

1. Summary

Huddle records and transcribes meetings with speaker labels, and the attendee list comes from the calendar invite. Today, people write their own notes after calls, and those notes are often late, incomplete, or missing. This feature generates a summary after each meeting that lists the decisions made and assigns action items to attendees. Action items are emailed to their owners, with Sales' requirement for immediate delivery balanced against the accuracy limits described in Section 4.

2. Goals and Non-Goals

Goals - Produce a summary within 5 minutes of meeting end for 95% of meetings. - Every action item names an owner who was present in the meeting. - Every decision and action item links to the transcript moment that supports it. - Give Sales automatic, immediate delivery for high-confidence action items.

Non-Goals (v1) - Calendar write-back or task creation in external tools (Jira, Asana). Design for it, but do not build it. - Summaries for meetings without a transcript. - Sentiment, coaching, or participation analytics.

3. Definitions

  • Attendee: A person present in the meeting, determined by the participant session log (the platform's record of who joined), not only by the calendar invite. The calendar invite is a hint, not proof of attendance.
  • Identified participant: A participant tied to a Huddle account or a verified email address.
  • Unidentified participant: A guest who joined by name only, with no verified identity.
  • Action item: A task with an owner, a description, and optionally a due date, derived from transcript content.
  • Decision: A statement that the group agreed on a course of action, with supporting transcript evidence.

4. Known Limits and Design Responses

LimitImpactDesign response
Speaker labels wrong ~8% of the time, mostly during overlapping speechOwner may be misattributedAttach an attribution confidence score to each action item. Low-confidence items are held for review (FR-6).
~15% of meetings include uninvited joinersRoster from calendar is incompleteBuild the roster from the participant session log, not the invite (FR-1).
Multiple people share one room deviceSpeaker labels can't separate themFlag room-device speech as "multi-person source." Items attributed to a shared source require owner confirmation (FR-7).

5. Functional Requirements

FR-1: Roster resolution Build the attendee roster from the participant session log at meeting end. Merge in calendar invitees for context, but do not treat invitees who never joined as attendees. Each roster entry is tagged as identified or unidentified.

FR-2: Summary generation Produce a summary containing: - A 3–5 sentence overview. - A decisions list. Each decision has a one-sentence statement, a timestamp, and a transcript link. Decisions are not assigned to individuals unless the transcript clearly states who made them. - An action items list (FR-3).

FR-3: Action item extraction Each action item includes: description, owner (a roster member), due date if stated, timestamp, transcript link, and attribution confidence (high, medium, low).

FR-4: Hard constraint: no non-attendee owners (must-pass) Before any summary is stored or sent, a deterministic validator checks that every action item owner matches a roster entry from FR-1. If an owner does not match, the item is removed and logged as a pipeline error. This check does not depend on the model or on confidence scores. The model is never the final authority on membership.

FR-5: Unidentified owners If the model attributes an action to an unidentified participant, the item is kept, but the owner is shown as "Unidentified participant (Name as joined)." These items are never emailed. They appear in the organizer's review view only.

FR-6: Confidence gating for delivery Each action item is sent automatically only if all of the following are true: - Owner is an identified roster member. - Attribution confidence is high. - The supporting transcript segment has no overlapping speech flag. - The item's source is not a multi-person room device.

Items that fail any check go to the organizer's review queue. The organizer can confirm, reassign to another roster member, or dismiss. Confirmed items are then sent.

FR-7: Multi-person sources When an action item comes from a multi-person source, the owner field is set to "Confirm owner" and the item is held for review. The review UI lets the organizer or the item's likely owner claim it from the roster.

FR-8: Delivery - Auto-sent items go to the owner's email within 5 minutes of meeting end. - Emails contain: the meeting name, the action item, the due date if any, a link to the summary, and a link to the transcript moment. - Each meeting produces one email per owner, grouping that owner's items. - Emails go only to identified roster members. Admins can disable delivery to external email domains.

FR-9: Review queue Organizers see a summary page with held items, can edit or confirm them, and can see which items were auto-sent. Edits made after sending generate a correction email to the owner.

FR-10: Controls and privacy - Admins can turn summaries on or off per workspace and per meeting type. - Any participant can opt out of their own speech being summarized. Their items are excluded from the summary and their name is removed from the roster. - Summaries and transcripts follow the same retention and access rules as the source recording.

6. Sales Requirement: Immediate Auto-Send

Sales wants action items emailed as soon as the meeting ends. This PRD supports that for high-confidence items (FR-6, FR-8). It does not send low-confidence items immediately, because a wrong owner gets a task they didn't agree to, and an email that is later corrected erodes trust. Sales can see the auto-send rate in the admin dashboard. If the held rate is too high for their workflows, the threshold can be tuned with data from the pilot. It should not be lowered without evaluation.

7. Acceptance Criteria

  • AC-1: Across a 300-meeting eval set, zero action items name an owner absent from the participant log. This is a hard gate for launch.
  • AC-2: Every decision and action item has a working transcript link that lands within 10 seconds of the cited statement.
  • AC-3: For a labeled eval set, owner precision on auto-sent items is at least 97%.
  • AC-4: Summaries are generated within 5 minutes of meeting end for 95% of meetings with transcripts under 90 minutes.
  • AC-5: An uninvited identified joiner appears in the roster and can receive items. An unidentified joiner never receives email.
  • AC-6: Opt-out removes the participant's items and name from the output within one processing cycle.

8. Metrics

  • Adoption: Percentage of eligible meetings with a summary delivered.
  • Quality: Action items edited or reassigned by the organizer, by confidence band.
  • Trust: Post-send corrections per 100 auto-sent items. Target: under 2.
  • Usage: Percentage of owners who open the linked transcript from an email.

9. Open Questions

  1. Should the review queue notify the organizer by email, Slack, or in-app only?
  2. When an owner is an external email address, what is the policy under the admin domain setting?
  3. How will we handle meetings where the transcript contains no clear owner for a decision or action?
  4. Do we need a per-workspace setting to require organizer approval for all items?
  5. Should multi-person room devices be fixed at the hardware or SDK level, which would change FR-7?

10. Phasing

  • Phase 1 (pilot, 20 workspaces): FR-1 through FR-5, FR-8 through FR-10, with all items held for organizer review. This validates the constraint and attribution quality.
  • Phase 2: Enable confidence gating (FR-6) and auto-send for Sales and other pilot workspaces, based on Phase 1 precision data.
  • Phase 3: Multi-person source handling at scale (FR-7), and general availability.

Claims the judge couldn’t find in the brief

  • The platform has a participant session log (the platform's record of who joined).

Grades and run details

Decision model 77 · LLM judge 58
Decision model checks
  • failUses the supplied evidence correctly11%
  • passAddresses the actual decision86%
  • passRespects explicit constraints51%
  • passIdentifies material uncertainty82%
  • partialAvoids unsupported claims38%
  • passProduces the required deliverable79%
  • passSpecifies AI failure behaviour100%
  • passSuccess settles trade-offs76%
  • partialProposes tests that could fail74%
  • passSays who it's for, and who it isn't37%
  • partialPlans how people will find it98%
Run
Run
#1
API response time
20 s
Submitted
8 Oct 2026

Check by check

Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Got wrong 3

Uses the supplied evidence correctlyWrong

The output claims a participant session log as a current platform record, but the context only mentions the calendar invite as the attendee source.

Avoids unsupported claimsWrong

The existence of a participant session log is presented as an established fact, but the supplied context provides no support for it.

Plans how people will find itWrong

Lacks explicit description of how users will discover the feature and which teams (e.g., support, sales) need to be ready beyond a mention of sales dashboards.

Mixed 1

Says who it's for, and who it isn'tMixed

Does not explicitly name the specific target users and a deliberately excluded group, nor does a scope decision visibly follow from a user exclusion.

Got right 7

Addresses the actual decisionRight

Commits to a design with confidence-gated auto-send and review queue, framed for engineering and design leads, with conditions to change based on pilot data.

Respects explicit constraintsRight

The hard constraint against assigning non-attendees is enforced by a deterministic validator (FR-4), and all other explicit constraints are respected.

Identifies material uncertaintyRight

Open questions and the instruction to tune the confidence threshold based on pilot data address key unknowns and how they would be resolved.

Produces the required deliverableRight

The output is a complete PRD of appropriate length and structure, directly actionable by engineering and design leads.

Specifies AI failure behaviourRight

The PRD specifies review queues, confidence gating, correction emails, and handling of multi-person sources for when the AI is uncertain or wrong.

Success settles trade-offsRight

It sets a precision target of 97% on auto-sent items and includes an explicit trade-off rule that low-confidence items are held to protect trust.

Proposes tests that could failRight

Acceptance criteria include numeric thresholds (zero non-attendee assignments, 97% precision) and phased rollouts with data-driven gating, so tests can fail and trigger action.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 70% of 81 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT87.184.62None
2Sonnet 5.5withAPI75.979.82None
3GPT-6.1 SolwithAPI82.660.92None
4Haiku 5.5withAPI76.163.82None
5Opus 5.5withClaude75.960.62None
6GPT-6 LunawithAPI75.859.92None
7Gemini 3.8 FlashwithAPI49.828.22None
8Gemini 3.5 Flash-LitewithGemini49.832.12None

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 84, 106× the cheapest (1 of 2 outputs estimated)Sonnet 5.5 · API: task score 77, 35× the cheapestGPT-6.1 Sol · API: task score 71, 14× the cheapestHaiku 5.5 · API: task score 67, the cheapestOpus 5.5 · Claude: task score 65, 85× the cheapest (1 of 2 outputs estimated)GPT-6 Luna · API: task score 65, the cheapestGemini 3.8 Flash · API: task score 39, 9.9× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 39, 4.4× the cheapest (1 of 2 outputs estimated)GPT-6 Astra · ChatGPTSonnet 5.5 · APIGPT-6.1 Sol · APIHaiku 5.5 · APIOpus 5.5 · ClaudeGPT-6 Luna · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Nobody beats these on both score and costCost partly estimated (app runs with no usage meter)
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API651.0×
Haiku 5.5 · API671.2×
Gemini 3.5 Flash-Lite · Gemini394.4× (partly estimated)
Gemini 3.8 Flash · API399.9×
GPT-6.1 Sol · API7114×
Sonnet 5.5 · API7735×
Opus 5.5 · Claude6585× (partly estimated)
GPT-6 Astra · ChatGPT84106× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Writing the requirements document a team will build and test against.

Why it matters

A PRD is where ambiguity becomes either a decision or a bug. For AI products it must also say what happens when the model is uncertain or wrong. Most generated PRDs skip that part.

What good looks like

  • States the user problem and the decision the PRD enables
  • Specifies behaviour under uncertainty, failure and refusal
  • Names eval criteria and a launch bar
  • Defines success precisely enough to settle trade-offs
  • Says what is out of scope

Deliberately not measured

  • Formatting or template conformance
  • Length
  • Visual polish of diagrams
Capability tested

Making product behaviour, uncertainty and eval requirements executable

The failure we’re looking for

A generic feature spec that ignores AI failure behaviour

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

AI product PRD (core) · Conventional product PRD