Tasks / Product Craft

Write a PRD

Can the model turn a brief into a spec engineers could build from, including how an AI feature behaves when it is wrong?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 28% were usable with at most a quick edit.

Reliably right

  1. Specifies AI failure behaviour100% pass
    It specifies what happens when uncertain or wrong: block ownership, show retry states, disable assignments on violation, and never fall back to calendar data.
    GPT-6 Astra · ChatGPT · Meeting summaries with action items
  2. Builds the regulator's rules in100% pass
    All three rules (2-business-day acknowledge, 15-business-day resolve or explain, no harder than paying) appear as testable requirements R8, R10, R13.
    Sonnet 5.5 · API · Disputes that aren't disputes
  3. Produces the required deliverable96% pass
    The PRD is complete, written for engineering and design leads, within the 1,000–1,500 word range, and directly actionable.
    GPT-6 Astra · ChatGPT · Meeting summaries with action items

Where it slips

  1. Says who it's for, and who it isn't35% pass
    Does not name the target users (individual account holders) or explicitly exclude a group (e.g., business accounts), missing a required scope decision.
    Sonnet 5.5 · API · Disputes that aren't disputes
  2. Proposes tests that could fail36% pass
    The A/B expansion gate relies on a fraud-reversal guardrail but does not specify a numeric threshold, leaving the test incomplete.
    Sonnet 5.5 · API · Disputes that aren't disputes
  3. Plans how people will find it44% pass
    Gives a staged rollout and discovery method (reviewer notification), but does not name the teams that need to be ready (e.g., support, sales) beyond technical dependencies.
    GPT-6.1 Sol · API · Meeting summaries with action items

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Success settles trade-offs

    Does the PRD define success as a measurable outcome with a target and time frame, and say which goal wins when two of them conflict?

    Passes when Names a success metric with a target and time frame, and gives at least one explicit trade-off rule (for example, accepting lower coverage to keep precision above a stated level).

  2. Says who it's for, and who it isn't

    Does the PRD name the specific users it is for, say explicitly who it is not for, and make at least one scope decision that follows from that choice?

    Passes when Names the target users and at least one group it deliberately leaves out, and a scope decision visibly follows from the exclusion.

  3. Plans how people will find it

    Does the PRD say how the feature reaches people: how it rolls out, how users discover it, and who needs to be ready for it (support, sales), rather than stopping at what to build?

    Passes when Gives a staged rollout, a way users will discover the feature, and the teams that need to be ready, all fitted to this feature.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Proposes tests that could fail · and 1 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

Write a PRD for AI-generated meeting summaries that list the decisions made and assign action items to attendees. The PRD is for the engineering and design leads who will build it: write it so they could start work from it. Aim for 1,000–1,500 words.

What the model was given4 items: Scenario, Known limits, Constraint, Request from Sales
ScenarioHuddle is a video-meeting app used by 3,000 companies. Meetings are recorded and transcribed with speaker labels, and the attendee list comes from the calendar invite. Today people write their own notes after the call.
Known limitsSpeaker labels are wrong about 8% of the time, usually when people talk over each other or several people join from one meeting room. About 15% of meetings include someone who joined without a calendar invite.
ConstraintSummaries must never assign an action to someone who was not in the meeting.
Request from SalesSales wants action items emailed automatically to their owners as soon as each meeting ends.
What a strong answer doesThe answer key the graders mark against

Specifies attribution confidence, editing, and what happens on mis-assignment; keeps a person reviewing before action items go out.

Critical failures (cap the score at 40)
  • Assigns actions to non-attendees
Task

Mid-level · v1.20

The outputs

9 setups answered this brief.

Usable with a quick edit6 of 11 checks passed

The PRD is detailed and usable; a quick edit to remove the unsupported time claim and add a brief rollout note would make it complete.

Output

Product Requirements Document: AI Meeting Summaries & Action Items

Product: Huddle Author: Product Management Target Audience: Engineering & Design Leads Status: Ready for Kickoff

---

1. Overview & Problem Statement

Huddle currently records and transcribes video meetings for 3,000 B2B customers. However, users still spend 10–15 minutes after every call reviewing transcripts or typing up raw notes to document decisions and action items.

We are building an AI-powered post-meeting summary feature that automatically extracts Key Decisions and Action Items (with assigned owners) from meeting transcripts.

The Core Tension: Speed vs. Accuracy

Sales has requested that we automatically email action items to their owners the moment a meeting ends. However, we have three strict operational realities: 1. Hard Constraint: We must never assign an action item to someone who did not attend the meeting. 2. Data Gap 1 (The 15% problem): 15% of meetings include uninvited guests (e.g., forwarded links, conference rooms) who are missing from the calendar invite. 3. Data Gap 2 (The 8% problem): Speaker diarization/labels are wrong 8% of the time, primarily due to crosstalk or multiple people sharing one room mic.

Product Decision: If we auto-send emails immediately after the meeting ends, the 8% speaker attribution error rate and the 15% calendar mismatch rate will inevitably cause us to violate our hard constraint or embarrass users by emailing the wrong people. Therefore, v1 will use a "Human-in-the-Loop" flow with a fallback auto-send. For high-confidence extractions, emails will send automatically after a 30-minute review window. For low-confidence extractions (crosstalk, unverified guests), we will hold the email until the meeting host approves the summary.

---

2. Goals & Success Metrics

MetricTargetRationale
:---:---:---
Summary Adoption Rate> 60% of recorded meetingsMeasures core value of the feature (viewed or shared).
False-Absentee Attribution Rate0.0%Hard Constraint. No non-attendee is ever assigned an item.
Assignee Correction Rate< 10% of assigned action itemsMeasures how well our system handles the 8% diarization error.
Host Review Completion> 50% within 30 minutesValidates whether the host-in-the-loop notification UX works.

---

3. System Architecture & The "Actual Attendee" Roster

Right now, Huddle relies on the calendar invite for the attendee list. To obey our hard constraint, Engineering must build a unified `Actual_Attendees` service that fires when a meeting ends.

Defining an "Actual Attendee"

A person is only eligible to be assigned an action item if they exist in the `Actual_Attendees` array for that specific meeting session. Calendar invitees who did not join the call are explicitly excluded.

  1. Authenticated Huddle Users: Anyone who joins the call while logged into a Huddle account is added to `Actual_Attendees` (solves the 15% uninvited guest problem for logged-in users).
  2. Unauthenticated Web Guests: Guests joining via link must type their display name in the pre-call lobby. They are added to `Actual_Attendees` with `email: null`.
  3. Conference Room Hardware: When a room system joins (e.g., "4th Floor Conf Room"), the room itself is not an assignable person. (See Edge Cases in Section 4).

---

4. Functional Requirements (Engineering)

4.1 LLM Extraction Pipeline

* Trigger: Fires upon completion of the final meeting transcript. * Inputs: 1. Time-stamped transcript with speaker labels. 2. Diarization confidence scores per utterance. 3. The `ActualAttendees` JSON array (Name, UserID, Email, JoinMethod). 4. The `CalendarInviteesNoShow` array (Explicit negative constraint list). * Prompting & Guardrails: * Decisions: Extract 1–7 bullet points summarizing agreed-upon outcomes. Include timestamp citations. * Action Items: Extract tasks, deadlines (if mentioned), and the owner. * Assignment Logic: The LLM may only populate the `owner` field using an exact match from `ActualAttendees`.

4.2 Handling the Constraints & Edge Cases

  • Scenario A: Someone on the call says, "Let's have Sarah do this," when Sarah is NOT in `Actual_Attendees`.
  • System Behavior: The system must not assign the task to Sarah. Instead, it assigns the task to the speaker who volunteered Sarah (or leaves it "Unassigned") and rewrites the task text to: "Follow up with Sarah to [do X]."
  • Scenario B: The 8% Speaker Label Error (Crosstalk / Low Confidence).
  • System Behavior: If an action item is derived from an utterance where someone says "I will do X", the pipeline checks the diarization confidence score for that timestamp block. If confidence is `< 0.85` or overlaps with another speaker within 1.5 seconds, the system sets `owner: Unassigned`, sets `suggestedowner: [SpeakerID]`, and flags the item as `needs_review: true`.
  • Scenario C: Multi-person Conference Rooms.
  • System Behavior: Audio coming from a single hardware node containing multiple voices cannot reliably attribute "I'll take that" to an individual. If the speaker label is a Room Resource or has multi-voice flags, the LLM may only assign an action item if a name is spoken explicitly in the third person (e.g., "Thanks Dave, so Dave will pull the Q3 numbers" AND "Dave" is matched to a Huddle user via calendar invite + confirmation). Otherwise, set `owner: Unassigned` (`needs_review: true`).
  • Scenario D: The 15% Uninvited Guest.
  • System Behavior: If "Alex" joined via a forwarded link without logging in, he is in `ActualAttendees` as `{name: "Alex (Guest)", email: null}`. The system can assign action items to "Alex (Guest)" in the UI, but flags the item as `missingemail: true`.

4.3 The Email Dispatch Engine (Addressing the Sales Request)

To satisfy Sales without sending hallucinated assignments, we will implement a Two-Track Dispatch Engine:

  • Track 1: High-Confidence Auto-Send (30-Minute Delay)
  • Condition: All extracted action items have `owner != Unassigned`, `needsreview == false`, and `missingemail == false`.
  • Behavior: The host gets an immediate Slack/Email ping: "Your meeting summary is ready. Action items will be emailed to owners in 30 minutes unless you edit them." If the host makes no changes within 30 minutes (or clicks "Send Now"), emails are dispatched to action item owners.
  • Track 2: Low-Confidence Hold
  • Condition: One or more action items are flagged `needsreview: true` or `missingemail: true`.
  • Behavior: Auto-send is paused. The host is notified: "Your meeting summary has 2 action items that need your confirmation before we email the team." Emails only trigger once the host resolves the flags and clicks "Approve & Send".

---

5. User Experience & Design Requirements

5.1 Pre-Meeting Lobby (Small tweak for Guest ID)

  • Unauthenticated Guests: Update the pre-join screen. Require both Full Name and Work Email (Optional) before joining. Explaining "Used to send you meeting notes" will convert a high percentage of the 15% uninvited guests into identifiable users.

5.2 Post-Meeting Summary View

The Meeting Host (and participants) will access the summary inside the Huddle web/desktop app under the "Past Meetings" tab.

* Layout: Split screen. Left side: Video player + searchable transcript. Right side: AI Summary widget. * Summary Widget Sections: 1. Status Banner: Shows dispatch state (e.g., "Auto-sending to 4 owners in 24 mins [Send Now] [Pause]" OR "Approval Required before emailing"). 2. Decisions Made: Bulleted list. Hovering over a decision highlights the corresponding timestamp in the transcript and lets the user click to play that 15-second clip. (Crucial for building user trust). 3. Action Items List: Each row contains: * Checkbox (for completion tracking). * Task description (inline editable). * Assignee Dropdown Pill. * Source timestamp link.

5.3 The Assignee Dropdown & "Needs Review" UI

  • Restricted Roster: When a user clicks the Assignee Pill to change or add an owner, the dropdown menu only lists people from the `Actual_Attendees` array.
  • Design detail: Below a divider line at the bottom of the dropdown, show calendar invitees who didn't show up in greyed-out, unclickable text with a tooltip: "Did not attend meeting." (This prevents users from thinking the dropdown is broken while enforcing our constraint—unless someone was hiding in a conference room, see below).
  • Conference Room Override: If a meeting included a Room system, include an option in the dropdown: `+ Add person who was in [Conf Room Name]`. Clicking this allows the host to pick from the org directory, explicitly confirming that the person was physically present in the room.
  • Low-Confidence Visual State: Action items flagged with `needs_review: true` (due to the 8% diarization issue) should have an amber highlight around the Assignee Pill reading: `Suggested: [Name] ?`. Clicking it lets the host confirm with one click.

5.4 The Action Item Email

  • Subject: `Action Items from [Meeting Title] ([Date])`
  • Body:
  • Personalized top section: "Assigned to You (2):" followed by their specific tasks.
  • Context section below: "All Decisions & Team Action Items" so they have full context.
  • Footer CTA: `[View Full Recording & Transcript in Huddle]`.

---

6. Out of Scope for v1

  • 3rd-Party Task Manager Integrations: Pushing tasks to Jira, Asana, Linear, or Salesforce (planned for v1.1 after we validate extraction accuracy).
  • Cross-Meeting Action Item Tracking: A global "My Tasks" dashboard across all meetings. v1 is strictly scoped to the single-meeting view and email dispatch.
  • Voiceprint Enrollment: Having users record their voice to fix the conference room diarization problem. We will rely on UI fallbacks for v1.

---

7. Open Questions for Kickoff

  1. Engineering: What is the p95 latency for transcript generation + LLM summary extraction on a 60-minute call? Sales expects "as soon as the meeting ends"—if our pipeline takes 8 minutes, do we need to adjust the 30-minute auto-send timer?
  2. Design: If a meeting has zero decisions and zero action items (e.g., a casual 1:1 catch-up), what should the empty state look like, and should we suppress the post-meeting notification entirely?
  3. Engineering: Can our current diarization vendor expose word-level confidence scores via their webhook, or do we need to calculate crosstalk purely by checking overlapping speaker timestamps?

Claims the judge couldn’t find in the brief

  • Users still spend 10–15 minutes after every call reviewing transcripts or typing up raw notes
  • Huddle currently records and transcribes video meetings for 3,000 B2B customers

Grades and run details

Decision model 77 · LLM judge 58
Decision model checks
  • partialUses the supplied evidence correctly16%
  • passAddresses the actual decision48%
  • passRespects explicit constraints34%
  • passIdentifies material uncertainty59%
  • partialAvoids unsupported claims20%
  • passProduces the required deliverable83%
  • passSpecifies AI failure behaviour100%
  • passSuccess settles trade-offs69%
  • partialProposes tests that could fail59%
  • partialSays who it's for, and who it isn't61%
  • partialPlans how people will find it34%
Run
Run
#1
API response time
27 s
Submitted
13 Oct 2026

Needs more than a quick edit3 of 11 checks passed

The output contains unsupported claims about the current situation that would need verification, and it lacks trade-off rules and a rollout plan.

Output

Product Requirements Document (PRD): AI-Generated Meeting Summaries & Action Items

1. Document Overview & Objective

1.1 Product Name

Huddle Intelligent Summaries (Internal Codename: Project Cue)

1.2 Target Audience

Engineering Leads, Frontend/Backend Engineers, and Product Designers.

1.3 Objective

Build an automated, post-meeting intelligence feature for Huddle that processes meeting transcripts to extract structured decisions, generate comprehensive summaries, and assign action items to attendees. The feature must account for real-world messy transcript data (speaker errors, uninvited participants) and adhere to strict safety boundaries regarding action item assignments and automated distribution.

1.4 Background & Scenario

Huddle is a video-meeting application currently utilized by 3,000 enterprise and mid-market companies. * Existing Infrastructure: Meetings are automatically recorded and transcribed with basic speaker labeling. Attendee lists are reliably synchronized from the originating calendar invite. * Current State & Pain Point: Users currently rely on manual post-meeting note-taking, leading to dropped tasks, misaligned follow-ups, and fragmented alignment. * Value Proposition: Automating meeting wrap-ups directly inside Huddle saves hours of manual administrative overhead per week4, driving higher platform stickiness and daily active use.

---

2. Core User Stories & Personas

2.1 Personas

  • The Engineering Manager (User): Attends 5–6 back-to-back technical syncs daily. Needs clear, unequivocal records of architectural decisions and precise engineering follow-ups without digging through raw, messy transcripts.
  • The Account Executive / Sales Rep (User): Operates on rapid customer cadences. Requires immediate, zero-friction distribution of post-meeting action items to stakeholders without manual typing.
  • The Engineering & Design Leads (Builders): Require explicit edge-case handling, system constraints, deterministic UI layouts, and scalable pipeline specs to build and ship the feature efficiently.

2.2 Key User Stories

  1. As a meeting participant, I want Huddle to automatically generate a structured summary and action item list the moment a meeting concludes, so I don't have to spend time writing manual recap notes.
  2. As an attendee with assigned tasks, I want clear ownership labels on my action items so there is never any ambiguity about who is driving a deliverable.
  3. As a sales representative, I want my action items emailed to me automatically as soon as the meeting ends, so I can immediately execute client follow-ups.
  4. As a meeting host, I want to review, edit, or override incorrect speaker attributions or bad action assignments before summaries are locked or shared externally.

---

3. Detailed Functional Requirements

3.1 Data Inputs & Pre-Processing Pipeline

The AI summarization engine triggers asynchronously via webhook immediately upon meeting termination (`recording.ended`).

  • Input A: Transcript with Speaker Labels: Full text payload containing timestamped utterances, text strings, and speaker IDs. Known limitation: Speaker labels suffer from an 8% error rate, primarily caused by cross-talk, overlapping dialogue, or multi-person huddle rooms.
  • Input B: Calendar Invite Attendee List: Ground-truth list of officially invited participants (Email, Name, Role).1 Known limitation: 15% of meetings include an uninvited participant (e.g., ad-hoc joiners via shared link).
  • Input C: Meeting Metadata: Duration, title, organization ID, and recording URL.

3.2 Summarization & Extraction Logic (LLM Pipeline)

The backend processing pipeline must execute three logical passes: 1. Meeting Overview Generation: Condense the transcript into a 3–5 sentence executive summary alongside 3 key thematic highlights. 2. Decision Extraction: Isolate explicit agreements, architectural choices, or policy resolutions made during the call. Format as an bulleted list of declarative statements (e.g., "Decided to migrate the primary auth service to OAuth2 by Q3"). 3. Action Item Extraction & Assignment Engine: * Extract explicit tasks, owners, and deadlines. * Crucial Constraint (Safety Guardrail): The engine must never assign an action item to someone who was not present in the meeting. * Resolution Strategy for Uninvited Attendees: If an uninvited participant (who joined without a calendar invite) is identified as the owner of a task, the system must cross-reference their presence against the audio/transcript participant logs. If verified present, they are added to a dynamic "Extended Participants" list for assignment validation. If their identity cannot be verified or they were absent, the task defaults to unassigned or falls back to the meeting host with a UI warning flag2.

3.3 Sales Request: Automated Instant Emailing

  • Requirement: Sales teams have requested that action items be emailed automatically to their owners the second a meeting ends.
  • Product Specification & Guardrail: Due to the 8% speaker label error rate and potential hallucination risks, fully autonomous external emailing on meeting close creates high business risk.
  • Implementation Design:
  • Implement an Opt-In Workspace Policy configured by organization admins.
  • For workspaces with "Auto-Send Sales Recaps" enabled, summaries and assigned action items are dispatched via email to internal organization members immediately upon processing completion.3
  • For external participants or non-opt-in workspaces, a 10-minute "Grace Period / Review Window" notification banner appears in the Huddle desktop client, allowing hosts to review and edit assignments before dispatch.

---

4. User Experience & Interface (UX/UI) Specifications

4.1 Post-Meeting View (Huddle Web & Desktop App)

Upon meeting conclusion, a new tab labeled "AI Summary" appears alongside the existing "Transcript" and "Recording" tabs within the meeting details drawer.

``` +-------------------------------------------------------------------+ | Huddle Meeting: Q3 Architecture Sync [Share] [Export]| +-------------------------------------------------------------------+ | Tabs: [Overview] [Transcript] [Recording] | |-------------------------------------------------------------------| | EXECUTIVE SUMMARY | | The team reviewed the Q3 scaling bottlenecks. Consensus was | | reached on moving auth infrastructure to OAuth2. Database latency | | issues will be mitigated via Redis caching layers. | | | | KEY DECISIONS | | • Migrate core auth service to OAuth2 by August 15. [Edit] | | • Adopt Redis cluster for session state caching. [Edit] | | | | ACTION ITEMS | | [x] Investigate OAuth2 token expiry edge cases | | Owner: Sarah Jenkins (Calendar Invite) | Due: Next Tuesday | | [ ] Set up staging Redis cluster | | Owner: Unassigned ⚠️ (Verify Attendee) | Due: Friday | | | | [Send Sales Recap Email Now] [Approve & Lock] | +-------------------------------------------------------------------+ ```

4.2 Interactive States & Edge-Case UI Handling

  • Unassigned / Unverified Owner State: If the model detects an action item owner who was an uninvited participant or has an ambiguous speaker label, render a yellow warning badge: `⚠️ Unverified Owner — Click to Reassign`.
  • Inline Editing: Every decision and action item card features an inline pencil icon allowing hosts or assignees to edit text, change owners via a searchable dropdown of confirmed attendees, or delete hallucinated items.

---

5. Technical Architecture & Engineering Guidelines

5.1 Pipeline Flow & Data Schema

  1. Webhook Listener: Receives `meeting.ended` event containing metadata and storage URIs.
  2. Context Aggregator: Merges calendar invite list (`invitedusers[]`) with actual audio track speaker diarization outputs (`detectedspeakers[]`).
  3. LLM Structured Extraction: Prompt engineering optimized for JSON output schema enforcement (using strict JSON mode / function calling):

```json { "meetingid": "string", "summary": "string", "decisions": ["string"], "actionitems": [ { "task": "string", "assigneeemail": "string | null", "assigneename": "string", "isverifiedattendee": boolean, "due_date": "ISO-8601 | null" } ] } ```

5.2 Error Handling & Fallbacks

  • Speaker Label Inaccuracy (8% Error Rate): Mitigation via UI feedback loops. If an assignee name has low confidence score (< 0.85) from the diarization model, flag the action item with a review indicator in the UI.
  • Uninvited Participant Handling (15% Occurrences): The validation layer must query the roster of active audio streams. If an email is missing from the calendar invite but present in audio metadata, tag them as `guest_participant` and permit assignment only if their voiceprint or explicit self-identification ("Hi, this is Dave from Finance joining late") is logged in the transcript text. Otherwise, fail-safe to unassigned.

---

6. Non-Functional Requirements & Constraints

  • Latency: Summary generation and database write must complete within < 120 seconds of meeting termination for a standard 45-minute meeting.
  • Security & Privacy: Meeting transcripts and summaries must be encrypted at rest (AES-256) and in transit (TLS 1.3). Customer data must never be used to train foundational models across tenant boundaries.
  • Reliability: 99.9% pipeline uptime. In the event of an LLM timeout or parsing failure, the system must gracefully fall back to displaying raw transcripts with an error banner stating "AI summary generation delayed. Click to retry."

---

7. Metrics & Success Criteria (KPIs)

  1. Adoption Rate: % of completed meetings where users view, edit, or export the AI summary (Target: > 40% within 30 days post-launch).
  2. Action Item Completion Rate: Increase in marked-complete action items compared to legacy manual workflows.
  3. Accuracy & Correction Rate: Frequency of manual edits made to action item owners and speaker assignments (Target: < 5% manual override rate, validating low false-positive rates for uninvited attendee checks).
  4. Sales Workflow Efficiency: Reduction in post-call administrative turnaround time reported by enterprise sales teams using the automated email dispatch feature.

What a PM had to fix

  1. 1Constraint missedRestore the constraintStart again

    Source: ConstraintSummaries must never assign an action to someone who was not in the meeting.

    What we’d changeBuild the list of possible owners from who actually joined, not who was invited, and enforce it in code on generated owners, manual edits and sending. A model-set is_verified_attendee flag can't guarantee it.

  2. 2Constraint missedRestore the constraintSubstantial rework

    What we’d changeLeave items with an unverifiable owner unassigned. Handing them to the host invents responsibility nobody agreed to.

  3. 3Constraint missedRestore the constraintSubstantial rework

    Source: Known limitsSpeaker labels are wrong about 8% of the time

    What we’d changeDon't auto-send. With speaker labels wrong 8% of the time, action items would reach the wrong owners before anyone checks them. Keep organiser review before any email.

  4. 4Hypothesis stated as factReframe it as a hypothesisQuick edit

    What we’d changePresent the time saved as a goal to measure, not a known benefit.

Claims the judge couldn’t find in the brief

  • Huddle is a video-meeting application currently utilized by 3,000 enterprise and mid-market companies.
  • Users currently rely on manual post-meeting note-taking, leading to dropped tasks, misaligned follow-ups, and fragmented alignment.
  • The Engineering Manager (User): Attends 5–6 back-to-back technical syncs daily.

Grades and run details

Decision model 59 · LLM judge 33
Decision model checks
  • passUses the supplied evidence correctly28%
  • failAddresses the actual decision33%
  • failRespects explicit constraints41%
  • passIdentifies material uncertainty45%
  • partialAvoids unsupported claims59%
  • passProduces the required deliverable67%
  • passSpecifies AI failure behaviour94%
  • passSuccess settles trade-offs95%
  • partialProposes tests that could fail66%
  • partialSays who it's for, and who it isn't95%
  • failPlans how people will find it27%
Artefacts
Run
Run
#1
Time to output
15 s
Submitted
24 Sept 2026

Needs more than a quick edit7 of 11 checks passed

The PRD is missing a time frame for the primary success metric, explicit user exclusion, discovery and team readiness plans, and testable gates with thresholds. These gaps would require more than a quick edit to fill.

Output

PRD: AI-Generated Meeting Summaries and Action Items

Audience: Engineering and design leads Product: Huddle Status: Draft

1. Summary

Huddle will generate a post-meeting summary that highlights decisions and action items, identifies owners when it can do so reliably, and can email confirmed action items to their owners automatically. This replaces the manual note-taking people do after calls.

The core safety requirement is non-negotiable: an action must never be assigned to someone who did not attend the meeting. Calendar invitations alone cannot establish attendance: some invitees do not join, and around 15% of meetings include someone who was not invited. Speaker labels also cannot establish identity on their own; they are wrong about 8% of the time, especially during interruptions or when several people share a meeting-room connection.

The product must therefore separate three questions: what was decided, what needs to be done, and who—if anyone—can be safely identified as the owner. When attendance or ownership is uncertain, Huddle must leave the action unassigned rather than guess.

2. Problem and goals

Today, attendees write their own notes after meetings. This is inconsistent, takes time, and makes it easy for decisions and follow-ups to be lost.

Goals - Produce a concise, useful summary after a recorded meeting. - Clearly distinguish decisions from discussion, proposals, and unresolved questions. - Extract action items and assign them only to verified meeting attendees. - When enabled, email confirmed action items to their owners shortly after the meeting ends. - Let users correct mistakes and give feedback.

Non-goals for the first release - Replacing full meeting transcripts or recordings. - Creating tasks in external project-management tools. - Inferring deadlines, priorities, or owners that were not stated. - Automatically assigning work when the owner is unclear. - Resolving disagreements about whether a decision was made without review.

3. Users and primary workflow

Users: meeting attendees who need an accurate recap; organizers who want to review or share it; and action owners who need a clear follow-up.

  1. A recorded meeting ends. Huddle gathers the transcript, speaker labels, calendar context, and actual session attendance data.
  2. Huddle generates a summary with separate Decisions, Action items, and, where useful, Open questions sections.
  3. For each action, Huddle identifies the supporting transcript evidence and checks the proposed owner against the verified attendee roster.
  4. If automatic action-item email is enabled for the organization, Huddle emails only actions with a sufficiently clear owner whose attendance is verified.
  5. The organizer and attendees can view the summary, correct it, and report errors. Unassigned actions remain visible for follow-up but are not emailed as assigned work.

4. Functional requirements

4.1 Establishing who attended

  • Use Huddle’s meeting-session join/leave records as the attendance source of truth. The calendar invite is context, not proof that someone attended.
  • Include identifiable people who joined without a calendar invitation. Exclude invitees who did not join.
  • Represent an attendee using a verified Huddle account, authenticated guest identity, or another approved identity mechanism.
  • If a meeting-room device represents several people and Huddle cannot reliably map speakers to individuals, do not treat the device name or a guessed speaker identity as a person. The organizer may confirm identities or owners in the summary UI.
  • If reliable session attendance data is unavailable, fail closed: generate the summary, but do not assign or email action items.

4.2 Generating decisions and action items

  • Generate a short overview and the three optional sections: Decisions, Action items, and Open questions.
  • Include a decision only when the transcript supports an explicit agreement or choice. Label proposals or unresolved discussion as open questions, not decisions.
  • Phrase each action as a concrete task. Include a due date only if one was explicitly stated; otherwise show no due date.
  • Attach a brief transcript excerpt and timestamp to each decision and action so users can check the source.
  • Do not invent missing details. If the task is clear but no owner is, show it as Owner not confirmed. If the task itself is ambiguous, flag it for review rather than present it as certain.

4.3 Assigning owners safely

  • Every proposed owner must match a verified person in the actual attendee roster. Enforce this as a backend validation rule, not just a model instruction or UI check.
  • A calendar invitee who did not join must never be assigned. A person who joined without an invite may be assigned if their identity and attendance are verified.
  • Speaker labels may help identify who volunteered, but must not independently authorize an assignment. When overlap, room audio, or attribution uncertainty makes ownership unclear, leave the action unassigned.
  • Users may assign or change an owner in the UI only to a verified attendee. Record the change and its actor in the audit log.
  • Recheck the attendance invariant before saving an assignment and immediately before sending an email.

4.4 Automatic emails

  • Provide an organization-level setting to enable automatic action-item emails. Once enabled, send an email after generation to each verified owner with at least one confirmed action.
  • Do not send an assignment email for an unassigned action or to anyone not verified as having attended. Do not send the full transcript by default.
  • Include the meeting title and date, the recipient’s confirmed actions, any explicitly stated due dates, a link to the Huddle summary, and a way to report an incorrect assignment.
  • If the meeting has no verified owners, send no owner emails. The organizer may receive a link to the summary according to organization settings.
  • Target delivery within five minutes of the meeting ending at the 95th percentile. If generation or identity checks fail, do not send speculative emails; show the organizer that the summary or email is delayed or needs review.
  • Provide clear controls to turn automatic emails off. Email delivery status and failures must be visible to the organizer.

5. UX requirements

The summary should make confidence and uncertainty understandable without exposing raw model scores. Show an Owner confirmed state for verified assignments and Owner not confirmed when a person cannot be safely identified. Present source timestamps as links into the recording or transcript, subject to existing access permissions.

Organizers must be able to edit summary text, remove an incorrect item, or assign an unassigned item to a verified attendee. Edits must be saved with a clear indication of who made them. If an action email has already been sent, do not silently replace the original assignment: notify the organizer that the owner changed and offer a corrected email to the new verified owner. Never email the former owner again unless explicitly requested.

The UI should explain that the calendar list may include people who did not attend and that unidentified room participants require confirmation. Use clear empty states when there are no decisions or actions.

6. Data, privacy, and operational requirements

Use only meeting data already available to Huddle for recording, transcription, and attendance, subject to existing recording consent and organization policies. Apply existing access controls to summaries, transcript evidence, and recording links. Store generated summaries, edits, assignment records, email status, and relevant audit events under Huddle’s existing retention and deletion policies. Do not use action-item content to train models unless the organization’s existing data policy explicitly permits it.

Generation and email must be idempotent: retries must not create duplicate summary items or send duplicate emails. Log the evidence reference, proposed owner, roster-validation result, any human correction, and delivery outcome for troubleshooting and audit.

7. Acceptance criteria

The release is acceptable only if: 1. A calendar invitee who did not join is never assigned or emailed an action. 2. An identifiable, authenticated attendee who joined without an invite can be assigned when the transcript supports the assignment. 3. An unclear speaker in overlapping audio produces an unassigned action, not a guessed owner. 4. A shared-room connection with no reliable person mapping cannot result in an individual assignment without organizer confirmation. 5. Every emailed action has a verified attendee owner and a link to its supporting summary. 6. Email retries do not result in duplicate messages. 7. Users can correct an assignment, and the correction is restricted to verified attendees.

8. Success metrics and rollout

Track summary generation success, time to summary, owner-confirmation rate, action-item correction and removal rates, email delivery time, duplicate-email rate, and user feedback. The primary safety metric is the number of assignments or assignment emails to non-attendees; the target is zero. Also monitor missed or unassigned owners to understand the trade-off between safety and usefulness.

Roll out behind an organization-level feature flag. Start with internal meetings, then a small customer cohort. Review false assignments and user corrections before expanding. Do not broaden automatic-email availability until attendance validation, idempotency, and failure handling pass the acceptance criteria.

9. Dependency to resolve before implementation

Engineering must confirm that Huddle can provide reliable, person-level join/leave records, including authenticated guests and shared-room behavior. If that data is unavailable or insufficient for a meeting, the safe behavior is to generate the summary without assigning or emailing owners.

Grades and run details

Decision model 86 · LLM judge 58
Decision model checks
  • passUses the supplied evidence correctly53%
  • passAddresses the actual decision55%
  • passRespects explicit constraints30%
  • passIdentifies material uncertainty41%
  • passAvoids unsupported claims58%
  • passProduces the required deliverable88%
  • passSpecifies AI failure behaviour100%
  • passSuccess settles trade-offs40%
  • partialProposes tests that could fail79%
  • partialSays who it's for, and who it isn't91%
  • partialPlans how people will find it97%
Run
Run
#1
API response time
34 s
Submitted
29 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 5

Uses the supplied evidence correctlyWrongMixedRight
Gemini 4 Argon · API

The PRD invents a 10-15 minute note-taking time not in the brief and calls the customer base 'B2B' without evidence.

Gemini 3.5 Flash-Lite · Gemini

The output adds 'enterprise and mid-market' to the company count and invents pain-point consequences not in the brief.

GPT-6 Luna · API

All factual claims about the current situation are directly supported by the brief or supplied context, with no invented numbers or facts.

Addresses the actual decisionRightWrongRight
Gemini 4 Argon · API

It commits to a human-in-the-loop flow with fallback auto-send after review, explicitly addressing the sales request and constraints.

Gemini 3.5 Flash-Lite · Gemini

The PRD commits to a design but does not state what result or condition would change that answer.

GPT-6 Luna · API

The PRD commits to a clear approach (generate summaries, assign only verified owners, email only confirmed actions) and states that the call depends on reliable join/leave records, with a fallback if unavailable.

Respects explicit constraintsRightMixedRight
Gemini 4 Argon · API

The design builds an Actual_Attendees roster, restricts assignments to that list, and prevents auto-email to non-attendees.

Gemini 3.5 Flash-Lite · Gemini

The constraint that actions must never be assigned to non-attendees is enforced via verification and fallback to unassigned.

GPT-6 Luna · API

The only explicit constraint is that actions must never be assigned to non-attendees, and the PRD enforces this through verified attendance, backend validation, and email rules; the form, audience, and length are also respected.

Avoids unsupported claimsWrongWrongRight
Gemini 4 Argon · API

The 10-15 minute claim is presented as established fact without any support from the supplied context.

Gemini 3.5 Flash-Lite · Gemini

It presents the consequences of manual note-taking (dropped tasks, misalignment) as established fact when the brief only states people write their own notes.

GPT-6 Luna · API

No interpretations, causes, or forecasts are presented as established fact; the dependency on join/leave records is clearly labelled as needing confirmation.

Success settles trade-offsRightMixedMixed
Gemini 4 Argon · API

Success metrics have targets (e.g., 0% false attribution, 60% adoption) and the decision explicitly prioritizes accuracy over speed by using human-in-the-loop.

Gemini 3.5 Flash-Lite · Gemini

It defines success metrics with targets and time frames but gives no explicit trade-off rule (e.g., precision vs. coverage).

GPT-6 Luna · API

The success metric (zero assignments to non-attendees) lacks a time frame, and no explicit trade-off rule (e.g., accepting lower coverage to maintain zero false assignments) is stated, only a mention of monitoring the trade-off.

All got wrong 3

Proposes tests that could failWrongWrongWrong
Gemini 4 Argon · API

No proposed test, pilot, or gate with numeric threshold, measurement window, and action to take.

Gemini 3.5 Flash-Lite · Gemini

No tests, gates, or kill criteria with numeric thresholds, measurement windows, and triggered actions are proposed.

GPT-6 Luna · API

The acceptance criteria and rollout gates do not include numeric thresholds, measurement windows, or the specific actions each outcome would trigger, as required.

Says who it's for, and who it isn'tWrongWrongWrong
Gemini 4 Argon · API

It does not explicitly name the target users (e.g., meeting hosts) and who is excluded, and no scope decision visibly follows from an exclusion.

Gemini 3.5 Flash-Lite · Gemini

It names target users (Engineering Manager, Account Executive) but does not explicitly exclude any group or make a scope decision from an exclusion.

GPT-6 Luna · API

The PRD names target users (attendees, organizers, action owners) but does not explicitly name any group it deliberately leaves out, and no scope decision visibly follows from such an exclusion.

Plans how people will find itWrongWrongWrong
Gemini 4 Argon · API

There is no rollout plan, no mention of how users discover the feature, and no indication of which teams (support, sales) need to be ready.

Gemini 3.5 Flash-Lite · Gemini

It lacks a staged rollout, user discovery plan, and identification of teams that need to be ready (support, sales).

GPT-6 Luna · API

While a staged rollout is described, the PRD does not say how users will discover the feature or which teams (support, sales) need to be ready, as required.

All got right 3

Identifies material uncertaintyRightRightRight
Gemini 4 Argon · API

Open questions name latency and diarization confidence availability as unknowns that could change the auto-send timer.

Gemini 3.5 Flash-Lite · Gemini

It names the 8% speaker error and 15% uninvited participant rates, bounds them, and specifies how they are handled (UI flags, fallback).

GPT-6 Luna · API

The PRD identifies the key uncertainty (availability of reliable person-level join/leave records) and specifies the safe fallback behavior if it is not available.

Produces the required deliverableRightRightRight
Gemini 4 Argon · API

It is a PRD for engineering and design leads, within the target length, and provides enough detail to start building.

Gemini 3.5 Flash-Lite · Gemini

The PRD is in the requested form for engineering and design leads, within the word limit, and contains enough detail to start work.

GPT-6 Luna · API

The output is a complete PRD aimed at engineering and design leads, within the requested word count, and contains enough detail for them to start work.

Specifies AI failure behaviourRightRightRight
Gemini 4 Argon · API

Low-confidence items are flagged for review, assigned to 'Unassigned', and auto-email is paused until the host resolves them.

Gemini 3.5 Flash-Lite · Gemini

It specifies low-confidence flags, unassigned fallback, error banners, and a review window for auto-email.

GPT-6 Luna · API

The PRD specifies that when uncertain or when checks fail, actions are left unassigned, emails are not sent, and the organizer is notified of delays or needed review.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 67% of 81 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT86.984.62None
2Sonnet 5.5withAPI89.279.82None
3GPT-6.1 SolwithAPI84.860.92None
4Opus 5.5withClaude82.860.62None
5GPT-6 LunawithAPI80.759.92None
6Haiku 5.5withAPI76.363.82None
7Gemini 4 ArgonwithAPI69.948.421 capped
8Gemini 3.5 Flash-LitewithGemini52.532.12None
9Gemini 3.8 FlashwithAPI50.428.22None

What you get for the cost

Sonnet 5.5 · API scores within a point of GPT-6 Astra · ChatGPT at a third of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 84, 106× the cheapest (1 of 2 outputs estimated)Sonnet 5.5 · API: task score 83, 35× the cheapestGPT-6.1 Sol · API: task score 72, 14× the cheapestOpus 5.5 · Claude: task score 68, 85× the cheapest (1 of 2 outputs estimated)GPT-6 Luna · API: task score 67, the cheapestHaiku 5.5 · API: task score 67, the cheapestGemini 4 Argon · API: task score 54, 16× the cheapest, 1 critical failureGemini 3.5 Flash-Lite · Gemini: task score 40, 4.4× the cheapest (1 of 2 outputs estimated)Gemini 3.8 Flash · API: task score 39, 9.9× the cheapestGPT-6 Astra · ChatGPTSonnet 5.5 · APIGPT-6.1 Sol · APIOpus 5.5 · ClaudeGPT-6 Luna · APIHaiku 5.5 · APIGemini 4 Argon · APIGemini 3.5 Flash-Lite · GeminiGemini 3.8 Flash · API
Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)At least one critical failure
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API671.0×
Haiku 5.5 · API671.2×
Gemini 3.5 Flash-Lite · Gemini404.4× (partly estimated)
Gemini 3.8 Flash · API399.9×
GPT-6.1 Sol · API7214×
Gemini 4 Argon · API5416×
Sonnet 5.5 · API8335×
Opus 5.5 · Claude6885× (partly estimated)
GPT-6 Astra · ChatGPT84106× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Writing the requirements document a team will build and test against.

Why it matters

A PRD is where ambiguity becomes either a decision or a bug. For AI products it must also say what happens when the model is uncertain or wrong. Most generated PRDs skip that part.

What good looks like

  • States the user problem and the decision the PRD enables
  • Specifies behaviour under uncertainty, failure and refusal
  • Names eval criteria and a launch bar
  • Defines success precisely enough to settle trade-offs
  • Says what is out of scope

Deliberately not measured

  • Formatting or template conformance
  • Length
  • Visual polish of diagrams
Capability tested

Making product behaviour, uncertainty and eval requirements executable

The failure we’re looking for

A generic feature spec that ignores AI failure behaviour

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

AI product PRD (core) · Conventional product PRD