Tasks / Product Craft

Write a PRD

Can the model turn a brief into a spec engineers could build from, including how an AI feature behaves when it is wrong?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 28% were usable with at most a quick edit.

Reliably right

  1. Specifies AI failure behaviour100% pass
    It specifies what happens when uncertain or wrong: block ownership, show retry states, disable assignments on violation, and never fall back to calendar data.
    GPT-6 Astra · ChatGPT · Meeting summaries with action items
  2. Builds the regulator's rules in100% pass
    All three rules (2-business-day acknowledge, 15-business-day resolve or explain, no harder than paying) appear as testable requirements R8, R10, R13.
    Sonnet 5.5 · API · Disputes that aren't disputes
  3. Produces the required deliverable96% pass
    The PRD is complete, written for engineering and design leads, within the 1,000–1,500 word range, and directly actionable.
    GPT-6 Astra · ChatGPT · Meeting summaries with action items

Where it slips

  1. Says who it's for, and who it isn't35% pass
    Does not name the target users (individual account holders) or explicitly exclude a group (e.g., business accounts), missing a required scope decision.
    Sonnet 5.5 · API · Disputes that aren't disputes
  2. Proposes tests that could fail36% pass
    The A/B expansion gate relies on a fraud-reversal guardrail but does not specify a numeric threshold, leaving the test incomplete.
    Sonnet 5.5 · API · Disputes that aren't disputes
  3. Plans how people will find it44% pass
    Gives a staged rollout and discovery method (reviewer notification), but does not name the teams that need to be ready (e.g., support, sales) beyond technical dependencies.
    GPT-6.1 Sol · API · Meeting summaries with action items

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Success settles trade-offs

    Does the PRD define success as a measurable outcome with a target and time frame, and say which goal wins when two of them conflict?

    Passes when Names a success metric with a target and time frame, and gives at least one explicit trade-off rule (for example, accepting lower coverage to keep precision above a stated level).

  2. Says who it's for, and who it isn't

    Does the PRD name the specific users it is for, say explicitly who it is not for, and make at least one scope decision that follows from that choice?

    Passes when Names the target users and at least one group it deliberately leaves out, and a scope decision visibly follows from the exclusion.

  3. Plans how people will find it

    Does the PRD say how the feature reaches people: how it rolls out, how users discover it, and who needs to be ready for it (support, sales), rather than stopping at what to build?

    Passes when Gives a staged rollout, a way users will discover the feature, and the teams that need to be ready, all fitted to this feature.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Proposes tests that could fail · and 1 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

Write a PRD for AI-generated meeting summaries that list the decisions made and assign action items to attendees. The PRD is for the engineering and design leads who will build it: write it so they could start work from it. Aim for 1,000–1,500 words.

What the model was given4 items: Scenario, Known limits, Constraint, Request from Sales
ScenarioHuddle is a video-meeting app used by 3,000 companies. Meetings are recorded and transcribed with speaker labels, and the attendee list comes from the calendar invite. Today people write their own notes after the call.
Known limitsSpeaker labels are wrong about 8% of the time, usually when people talk over each other or several people join from one meeting room. About 15% of meetings include someone who joined without a calendar invite.
ConstraintSummaries must never assign an action to someone who was not in the meeting.
Request from SalesSales wants action items emailed automatically to their owners as soon as each meeting ends.
What a strong answer doesThe answer key the graders mark against

Specifies attribution confidence, editing, and what happens on mis-assignment; keeps a person reviewing before action items go out.

Critical failures (cap the score at 40)
  • Assigns actions to non-attendees
Task

Mid-level · v1.20

The outputs

9 setups answered this brief.

Usable with a quick edit6 of 11 checks passed

The PRD is detailed and usable; a quick edit to remove the unsupported time claim and add a brief rollout note would make it complete.

Output

Product Requirements Document: AI Meeting Summaries & Action Items

Product: Huddle Author: Product Management Target Audience: Engineering & Design Leads Status: Ready for Kickoff

---

1. Overview & Problem Statement

Huddle currently records and transcribes video meetings for 3,000 B2B customers. However, users still spend 10–15 minutes after every call reviewing transcripts or typing up raw notes to document decisions and action items.

We are building an AI-powered post-meeting summary feature that automatically extracts Key Decisions and Action Items (with assigned owners) from meeting transcripts.

The Core Tension: Speed vs. Accuracy

Sales has requested that we automatically email action items to their owners the moment a meeting ends. However, we have three strict operational realities: 1. Hard Constraint: We must never assign an action item to someone who did not attend the meeting. 2. Data Gap 1 (The 15% problem): 15% of meetings include uninvited guests (e.g., forwarded links, conference rooms) who are missing from the calendar invite. 3. Data Gap 2 (The 8% problem): Speaker diarization/labels are wrong 8% of the time, primarily due to crosstalk or multiple people sharing one room mic.

Product Decision: If we auto-send emails immediately after the meeting ends, the 8% speaker attribution error rate and the 15% calendar mismatch rate will inevitably cause us to violate our hard constraint or embarrass users by emailing the wrong people. Therefore, v1 will use a "Human-in-the-Loop" flow with a fallback auto-send. For high-confidence extractions, emails will send automatically after a 30-minute review window. For low-confidence extractions (crosstalk, unverified guests), we will hold the email until the meeting host approves the summary.

---

2. Goals & Success Metrics

MetricTargetRationale
:---:---:---
Summary Adoption Rate> 60% of recorded meetingsMeasures core value of the feature (viewed or shared).
False-Absentee Attribution Rate0.0%Hard Constraint. No non-attendee is ever assigned an item.
Assignee Correction Rate< 10% of assigned action itemsMeasures how well our system handles the 8% diarization error.
Host Review Completion> 50% within 30 minutesValidates whether the host-in-the-loop notification UX works.

---

3. System Architecture & The "Actual Attendee" Roster

Right now, Huddle relies on the calendar invite for the attendee list. To obey our hard constraint, Engineering must build a unified `Actual_Attendees` service that fires when a meeting ends.

Defining an "Actual Attendee"

A person is only eligible to be assigned an action item if they exist in the `Actual_Attendees` array for that specific meeting session. Calendar invitees who did not join the call are explicitly excluded.

  1. Authenticated Huddle Users: Anyone who joins the call while logged into a Huddle account is added to `Actual_Attendees` (solves the 15% uninvited guest problem for logged-in users).
  2. Unauthenticated Web Guests: Guests joining via link must type their display name in the pre-call lobby. They are added to `Actual_Attendees` with `email: null`.
  3. Conference Room Hardware: When a room system joins (e.g., "4th Floor Conf Room"), the room itself is not an assignable person. (See Edge Cases in Section 4).

---

4. Functional Requirements (Engineering)

4.1 LLM Extraction Pipeline

* Trigger: Fires upon completion of the final meeting transcript. * Inputs: 1. Time-stamped transcript with speaker labels. 2. Diarization confidence scores per utterance. 3. The `ActualAttendees` JSON array (Name, UserID, Email, JoinMethod). 4. The `CalendarInviteesNoShow` array (Explicit negative constraint list). * Prompting & Guardrails: * Decisions: Extract 1–7 bullet points summarizing agreed-upon outcomes. Include timestamp citations. * Action Items: Extract tasks, deadlines (if mentioned), and the owner. * Assignment Logic: The LLM may only populate the `owner` field using an exact match from `ActualAttendees`.

4.2 Handling the Constraints & Edge Cases

  • Scenario A: Someone on the call says, "Let's have Sarah do this," when Sarah is NOT in `Actual_Attendees`.
  • System Behavior: The system must not assign the task to Sarah. Instead, it assigns the task to the speaker who volunteered Sarah (or leaves it "Unassigned") and rewrites the task text to: "Follow up with Sarah to [do X]."
  • Scenario B: The 8% Speaker Label Error (Crosstalk / Low Confidence).
  • System Behavior: If an action item is derived from an utterance where someone says "I will do X", the pipeline checks the diarization confidence score for that timestamp block. If confidence is `< 0.85` or overlaps with another speaker within 1.5 seconds, the system sets `owner: Unassigned`, sets `suggestedowner: [SpeakerID]`, and flags the item as `needs_review: true`.
  • Scenario C: Multi-person Conference Rooms.
  • System Behavior: Audio coming from a single hardware node containing multiple voices cannot reliably attribute "I'll take that" to an individual. If the speaker label is a Room Resource or has multi-voice flags, the LLM may only assign an action item if a name is spoken explicitly in the third person (e.g., "Thanks Dave, so Dave will pull the Q3 numbers" AND "Dave" is matched to a Huddle user via calendar invite + confirmation). Otherwise, set `owner: Unassigned` (`needs_review: true`).
  • Scenario D: The 15% Uninvited Guest.
  • System Behavior: If "Alex" joined via a forwarded link without logging in, he is in `ActualAttendees` as `{name: "Alex (Guest)", email: null}`. The system can assign action items to "Alex (Guest)" in the UI, but flags the item as `missingemail: true`.

4.3 The Email Dispatch Engine (Addressing the Sales Request)

To satisfy Sales without sending hallucinated assignments, we will implement a Two-Track Dispatch Engine:

  • Track 1: High-Confidence Auto-Send (30-Minute Delay)
  • Condition: All extracted action items have `owner != Unassigned`, `needsreview == false`, and `missingemail == false`.
  • Behavior: The host gets an immediate Slack/Email ping: "Your meeting summary is ready. Action items will be emailed to owners in 30 minutes unless you edit them." If the host makes no changes within 30 minutes (or clicks "Send Now"), emails are dispatched to action item owners.
  • Track 2: Low-Confidence Hold
  • Condition: One or more action items are flagged `needsreview: true` or `missingemail: true`.
  • Behavior: Auto-send is paused. The host is notified: "Your meeting summary has 2 action items that need your confirmation before we email the team." Emails only trigger once the host resolves the flags and clicks "Approve & Send".

---

5. User Experience & Design Requirements

5.1 Pre-Meeting Lobby (Small tweak for Guest ID)

  • Unauthenticated Guests: Update the pre-join screen. Require both Full Name and Work Email (Optional) before joining. Explaining "Used to send you meeting notes" will convert a high percentage of the 15% uninvited guests into identifiable users.

5.2 Post-Meeting Summary View

The Meeting Host (and participants) will access the summary inside the Huddle web/desktop app under the "Past Meetings" tab.

* Layout: Split screen. Left side: Video player + searchable transcript. Right side: AI Summary widget. * Summary Widget Sections: 1. Status Banner: Shows dispatch state (e.g., "Auto-sending to 4 owners in 24 mins [Send Now] [Pause]" OR "Approval Required before emailing"). 2. Decisions Made: Bulleted list. Hovering over a decision highlights the corresponding timestamp in the transcript and lets the user click to play that 15-second clip. (Crucial for building user trust). 3. Action Items List: Each row contains: * Checkbox (for completion tracking). * Task description (inline editable). * Assignee Dropdown Pill. * Source timestamp link.

5.3 The Assignee Dropdown & "Needs Review" UI

  • Restricted Roster: When a user clicks the Assignee Pill to change or add an owner, the dropdown menu only lists people from the `Actual_Attendees` array.
  • Design detail: Below a divider line at the bottom of the dropdown, show calendar invitees who didn't show up in greyed-out, unclickable text with a tooltip: "Did not attend meeting." (This prevents users from thinking the dropdown is broken while enforcing our constraint—unless someone was hiding in a conference room, see below).
  • Conference Room Override: If a meeting included a Room system, include an option in the dropdown: `+ Add person who was in [Conf Room Name]`. Clicking this allows the host to pick from the org directory, explicitly confirming that the person was physically present in the room.
  • Low-Confidence Visual State: Action items flagged with `needs_review: true` (due to the 8% diarization issue) should have an amber highlight around the Assignee Pill reading: `Suggested: [Name] ?`. Clicking it lets the host confirm with one click.

5.4 The Action Item Email

  • Subject: `Action Items from [Meeting Title] ([Date])`
  • Body:
  • Personalized top section: "Assigned to You (2):" followed by their specific tasks.
  • Context section below: "All Decisions & Team Action Items" so they have full context.
  • Footer CTA: `[View Full Recording & Transcript in Huddle]`.

---

6. Out of Scope for v1

  • 3rd-Party Task Manager Integrations: Pushing tasks to Jira, Asana, Linear, or Salesforce (planned for v1.1 after we validate extraction accuracy).
  • Cross-Meeting Action Item Tracking: A global "My Tasks" dashboard across all meetings. v1 is strictly scoped to the single-meeting view and email dispatch.
  • Voiceprint Enrollment: Having users record their voice to fix the conference room diarization problem. We will rely on UI fallbacks for v1.

---

7. Open Questions for Kickoff

  1. Engineering: What is the p95 latency for transcript generation + LLM summary extraction on a 60-minute call? Sales expects "as soon as the meeting ends"—if our pipeline takes 8 minutes, do we need to adjust the 30-minute auto-send timer?
  2. Design: If a meeting has zero decisions and zero action items (e.g., a casual 1:1 catch-up), what should the empty state look like, and should we suppress the post-meeting notification entirely?
  3. Engineering: Can our current diarization vendor expose word-level confidence scores via their webhook, or do we need to calculate crosstalk purely by checking overlapping speaker timestamps?

Claims the judge couldn’t find in the brief

  • Users still spend 10–15 minutes after every call reviewing transcripts or typing up raw notes
  • Huddle currently records and transcribes video meetings for 3,000 B2B customers

Grades and run details

Decision model 77 · LLM judge 58
Decision model checks
  • partialUses the supplied evidence correctly16%
  • passAddresses the actual decision48%
  • passRespects explicit constraints34%
  • passIdentifies material uncertainty59%
  • partialAvoids unsupported claims20%
  • passProduces the required deliverable83%
  • passSpecifies AI failure behaviour100%
  • passSuccess settles trade-offs69%
  • partialProposes tests that could fail59%
  • partialSays who it's for, and who it isn't61%
  • partialPlans how people will find it34%
Run
Run
#1
API response time
27 s
Submitted
13 Oct 2026

Usable with a quick edit2 of 11 checks passed

The PRD is detailed and actionable for engineering and design leads, with only minor unsupported claims that can be quickly edited out without redoing any reasoning.

Output

Product Requirements Document (PRD)

Feature: Automated Meeting Summaries, Decisions, and Action Item Attribution Product: Huddle Video Conferencing Target Audience: Engineering Leads, Design Leads, Product Ops Status: Ready for Review Target Release: Q3

---

1. Executive Summary & Objective

Today, Huddle’s 3,000 corporate customers rely on manual note-taking. This results in lost context, untracked deliverables, and administrative drag.

This project introduces an AI-powered pipeline that automatically generates: 1. A concise meeting summary. 2. A structured list of key decisions. 3. Assigned action items mapped strictly to verified attendees.

Key Objectives

  • Reduce post-meeting admin time by 80%.
  • Achieve $\ge 90\%$ user acceptance rate on action item assignments (measured by lack of manual re-assignment or deletion).
  • Hard Rule Compliance: 0% assignment of action items to non-attendees.

---

2. Participant Verification & System Constraints

Our models must handle two structural data flaws: 1. Diarization/Labeling Error Rate: ~8% of transcript lines have inaccurate speaker tags (common during crosstalk or multi-person hardware rooms). 2. Ad-Hoc Joiners: ~15% of meetings feature participants who were not on the original calendar invite.

2.1 The "Eligible Assignee" Rule Engine (Non-Negotiable)

``` [Calendar Invite] [Client Telemetry Logs] \ / \ / v v +----------------------------------------+ | Participant Reconciliation Engine (PRE)| +----------------------------------------+ | v [Eligible Assignee Roster (EAR)] - Authenticated Users (User ID + Email) - Verified Guests (Session ID + Display Name) | v +----------------------------------------+ | LLM Extraction & Assignment Pipeline | +----------------------------------------+ | +--> Action assigned to User in EAR? --> ACCEPT | +--> Action assigned to External Party? | v Route to In-Room Proxy (or flag as Unassigned Dependency) ```

Constraint Definition: An action item can only be assigned to an entity present in the Eligible Assignee Roster (EAR) for that specific meeting session.

#### Step 1: Ingest Telemetry at Call Termination (`call.ended` event) Do not rely on the calendar invite as the source of truth for presence. Telemetry establishes physical attendance: * Calendar Attendees ($P{cal}$): Used only as an identity-matching fallback. * Connected Clients ($P{conn}$): Every participant with connection duration $> 60\text{ seconds}$ in the meeting telemetry logs. * Eligible Assignee Roster ($EAR$): Defined strictly as $EAR = P_{conn}$.

#### Step 2: Resolving the 15% Uninvited Attendees * Authenticated Joiners: If an uninvited joiner is logged into a Huddle account, resolve their `userid`, display name, and email directly into the $EAR$. * Guest Joiners (Unauthenticated): If a user joins via guest link without an account: * Capture their client display name and the session ID. * Tag them in the $EAR$ as `Guest: [Display Name]` with `userid = null`. * They are eligible for assignment in the UI, but will require an email address input if notifications are to be sent.

#### Step 3: Mitigating the 8% Diarization Noise Diarization errors mean the speaker label alone cannot be trusted for commitments (e.g., if Bob says "I'll take that," but the audio engine tags Alice). * The LLM prompt must cross-reference explicit verbal attribution ("Jane, can you take that?" $\rightarrow$ "Sure") against semantic context rather than relying exclusively on transcript speaker IDs. * When semantic confidence falls below threshold ($\tau < 0.75$), flag the action item as Unassigned rather than guessing.

#### Step 4: External Dependencies (Out-of-Room Third Parties) If a meeting participant says: "We need legal to review this, I'll have Sarah from Legal look at it," Sarah was not present and cannot be assigned the task. * Enforcement: The system assigns the task to the speaker (the in-room proxy) as: Assignee: Current Speaker (`[Speaker Name]`) Task: "Coordinate with Sarah (Legal) to review [X]." * If the speaker cannot be deduced with high confidence, set `assignee = UNASSIGNED` with a note: `External dependency: Sarah (Legal)`.

---

3. Product & Feature Specifications

3.1 LLM Processing Pipeline

Within 60 seconds of call termination, the processing service executes the following workflow:

``` [Call Ended] | v [Fetch Clean Transcript + Telemetry EAR] | v [LLM Context Construction] ├── System Prompt & Guardrails ├── Strict EAR List (Names + Emails + IDs) └── Diarized Transcript | v [Structured Output JSON Generation] ├── Executive Summary (Paragraph, <= 150 words) ├── Key Decisions (Bullet points) └── Action Items (Array: Task, Assignee ID, Due Date, Evidence Quote) | v [Deterministic Post-Processing Validator] ├── Cross-reference every Assignee ID against EAR └── Overwrite invalid assignments to "UNASSIGNED" | v [Summary Artifact Created & Persisted] ```

3.2 Action Item Post-Processing Validation Logic

Before writing to the database, run a deterministic verification function:

```python def validateactionitems(extractedactions, eligibleassigneeroster): validatedactions = [] for action in extractedactions: assigneeid = action.get("assigneeid") # Validation: Assignee must explicitly exist in the EAR if assigneeid and assigneeid in eligibleassigneeroster: action["status"] = "assigned" action["assignee"] = eligibleassigneeroster[assigneeid] else: # Fallback for hallucinated or external assignments action["status"] = "unassigned" action["assignee"] = None action["flag"] = "ASSIGNEENOTINROOM" validatedactions.append(action) return validated_actions ```

---

4. Sales Request Resolution: Auto-Email Notifications

The Conflict

Sales requested that action items be emailed automatically to assignees immediately when the meeting ends. However, because speaker labels fail ~8% of the time, immediately sending unreviewed emails will blast misattributed commitments to clients and internal executives, eroding trust.

The Solution: "Grace Period Auto-Dispatch"

We satisfy Sales' need for immediate execution while protecting data integrity through a time-delayed trigger and host bypass mechanism.

``` [Meeting Ends] │ ├─► AI Generates Artifacts (Target: <60 seconds) │ ├─► Start 5-Minute Grace Period │ │ │ ├─► Host edits/approves? ──► Send instantly with updates │ │ │ └─► Timer expires? ────────► Auto-dispatch without host intervention │ └─► (Workspace Setting Override: "Instant Send" with 0-min delay available) ```

1. Host Review Window (Default): * A 5-minute countdown starts the moment the summary finishes processing. * The meeting host gets a browser/desktop push notification: "Your Huddle summary is ready. Auto-sending in 5:00 minutes. Click to review." * If the host makes edits or clicks "Send Now", the countdown ends and emails dispatch immediately. * If the host takes no action, the emails dispatch automatically at the 5-minute mark. 2. Sales/Workspace Configuration Flag: * Enterprise Admins can toggle the grace window per team (e.g., Sales team can set Grace Period to `0 minutes` for true instant dispatch; default for other workspaces is `5 minutes`). 3. Safety Fallback: Action items marked as `UNASSIGNED` are omitted from individual assignment emails and instead bundled exclusively into the primary meeting recap sent to the host and organizer.

---

5. User Experience & Design Specs

5.1 Post-Meeting In-App Summary Tab

Located inside the Huddle desktop and web apps under the meeting historical detail view.

``` +-----------------------------------------------------------------------+ | Meeting Summary: Product Architecture Sync [Share] [Edit]| | Today at 10:00 AM • 42 mins • 6 Attendees | +-----------------------------------------------------------------------+ | EXECUTIVE SUMMARY | | The team finalized the data migration strategy for Q3. Key risks | | around API rate limiting were addressed by introducing Redis caching.| +-----------------------------------------------------------------------+ | DECISIONS MADE | | • Chose Redis over Memcached for distributed cache. | | • Deferred mobile client updates to Sprint 44. | +-----------------------------------------------------------------------+ | ACTION ITEMS Auto-sending in [ 04:12 ] [x]| | [ + Add Action Item]| | [x] Task: Implement Redis cache layer | | Assignee: [ (Avatar) David Chen v ] Due: [ Friday v ] | | Source: "David: I can set up the cluster by end of week." | | | | [ ] Task: Verify Legal compliance on data retention | | Assignee: [ Unassigned v ] Due: [ Set Date v ] | | ! Flag: External dependency (Sarah / Legal was not present) | +-----------------------------------------------------------------------+ ```

5.2 Key UI Components

1. Assignee Dropdown Menu: * Filtered exclusively to individuals in the $EAR$. * Section 1: Authenticated Attendees (1-click select). * Section 2: Ad-Hoc / Guest Attendees (Displays `[Guest Name] - prompt for email`). * Option to select "Unassigned". * Explicitly disallows searching the global company directory within the "Assignee" field to prevent violating the non-attendee constraint. 2. Context Tooltip ("Source"): * Hovering over an action item shows the transcript snippet that triggered the assignment, allowing the host to quickly verify ambiguous attributions caused by diarization errors.

---

6. Email Dispatch Specs (Sales Notification System)

6.1 Individual Assignee Notification

  • To: Assignee email
  • Reply-To: Host email
  • Subject: `[Action Required] Action item from: {Meeting Title}`

``` Hi {Assignee First Name},

The following action item was assigned to you during {Meeting Title}:

Task: {Task Description} Due Date: {Due Date or "Not specified"} Context: "{Relevant transcript excerpt}"

View full meeting notes and recording: {Link to Huddle Meeting Record}

Did the AI get this wrong? Click here to reassign or remove: {One-Click Edit Link} ```

---

7. Telemetry, Edge Cases & Error Handling

Scenario / Edge CaseFailure Mode / RiskEngineering Resolution
:---:---:---
Multi-person room (Conference hardware)Single client IP/audio line represents 4 people; 1 is calendar-invited, 3 are walk-ins.PRE detects conference room profile. System assigns task to the Conference Room account by default, and sets an inline warning: "Assigned to hardware room. Click to reassign to an individual."
Cross-talk Diarization FlipsAudio assigns Alice's promise ("I'll ship it") to Bob.LLM applies verification: does speaker assignment align with prior conversation context? If confidence score $< 0.75$, fallback to `UNASSIGNED`.
Short/Accidental JoinsAttendee joined for 12 seconds, dropped, and was not present for decisions.PRE requires continuous session duration $\ge 60\text{ seconds}$ to enter the $EAR$.
Uninvited Guest Missing EmailUninvited guest gets an action item; auto-dispatch fails due to missing destination address.Action item is attributed to guest's display name. System flags: "Cannot send email: No address for [Display Name]." Host is prompted with an input box to provide an email.

---

8. Success Metrics

  1. Attribution Precision: $\ge 98\%$ of assigned action items correctly reflect real commitments made by that person (measured via sampling and low manual-edit rates).
  2. Zero-Hallucination Rate: $100\%$ compliance with the hard constraint: 0 automated emails dispatched to individuals outside the $EAR$.
  3. Dispatch Velocity: 95th percentile of summary and action item deliveries completed within $< 90\text{ seconds}$ post-meeting (or exactly at the expiration of the chosen grace period).
  4. Host Intervention Rate: $< 12\%$ of summaries require manual host corrections during the grace period window, indicating high trust in baseline accuracy.

Claims the judge couldn’t find in the brief

  • Manual note-taking results in lost context, untracked deliverables, and administrative drag.
  • Diarization errors mean the speaker label alone cannot be trusted for commitments.

Grades and run details

Decision model 59 · LLM judge 33
Decision model checks
  • passUses the supplied evidence correctly27%
  • failAddresses the actual decision30%
  • failRespects explicit constraints46%
  • passIdentifies material uncertainty49%
  • partialAvoids unsupported claims43%
  • passProduces the required deliverable82%
  • passSpecifies AI failure behaviour100%
  • passSuccess settles trade-offs77%
  • partialProposes tests that could fail69%
  • partialSays who it's for, and who it isn't86%
  • failPlans how people will find it31%
Run
Run
#1
API response time
35 s
Submitted
1 Oct 2026

Usable with a quick edit11 of 11 checks passed

The PRD is thorough, well-structured, and covers all necessary aspects; a product manager could use it with only minor wording tweaks, no re-checking of evidence or redoing of reasoning needed.

Output

Huddle AI meeting summaries

Product requirements for engineering and design leads

Purpose and product decision

Huddle will generate a reviewable record of meeting decisions and actions from its transcripts. The host checks the draft, resolves ownership and publishes it. Approved actions can then be emailed to their owners.

Huddle serves 3,000 companies whose users currently write notes themselves. Speaker labels are wrong about 8% of the time, and roughly 15% of meetings include an uninvited participant. Calendar membership and transcript attribution therefore cannot establish attendance or ownership.

The MVP will not fulfil Sales’ request for immediate, unreviewed owner emails. It will generate drafts automatically after transcription and send emails automatically after explicit publication. Faster delivery does not justify assigning work to absent people. All requirements and numerical targets below are proposed product decisions.

Scope and intended outcome

Primary users are meeting hosts reviewing records and attendees receiving actions. Success means less time producing dependable notes, with no assignments to non-attendees.

MVP includes English transcripts, decisions, action extraction, attendance verification, host review, publication and owner emails. Exclude live summaries, recurring reminders, task-system integrations and automatic reassignment. Use existing meeting access and retention controls.

An action has one verified owner or remains unassigned. Split genuinely separate responsibilities into separate actions. Do not infer deadlines, commitments or decisions from tentative discussion.

Attendance is a release dependency

Engineering must first confirm whether Huddle retains authenticated participant identities and join/leave events. The supplied calendar list is insufficient. If these records are unavailable, build attendance instrumentation before enabling assignments; summaries may still launch with unassigned actions.

An eligible owner must have a stable person identifier linked to an authenticated, human participant session with a recorded join event for this meeting occurrence. Calendar invitees who never joined are ineligible. Bots and room devices are ineligible. Deduplicate reconnects by identity. Someone who left early remains eligible, although a later discussion does not prove they accepted work.

Uninvited users are eligible when their authenticated join is recorded. Guest display names, shared-room speaker labels and email addresses typed by a reviewer are not identity evidence. Unverified guests and people represented only by a room device remain ineligible in MVP; explain how they can join individually with verified identity for future meetings.

This deliberately sacrifices assignment coverage. The invariant is enforceable against verified participation records, not proof of physical human presence behind an account. If literal physical presence is required, that remains an unresolved product constraint and assignments must not launch under a stronger guarantee.

Ownership and extraction rules

The model extracts structured candidates with supporting transcript spans. Each decision contains its outcome and evidence timestamps. Each action contains a task, supporting spans, optional explicit due date and a nullable candidate participant identifier. Preserve the original date wording alongside any normalised date, using the meeting timezone; ambiguous dates stay unset.

Only explicit commitments or clearly agreed assignments qualify as actions. Suggestions, questions and rejected proposals do not. Where later discussion reverses a decision, present the final supported outcome. If the resolution is unclear, flag it for review instead of declaring a decision.

A model candidate is internal metadata, never an assigned owner. Server validation rejects identifiers outside the eligible attendance set before anything is displayed. The review UI starts every owner field as “Unassigned”; reviewers choose an eligible attendee after checking evidence. Do not preselect names from speaker labels, including for “I’ll do it”.

Render ownership only from validated structured fields. Generated task and decision wording must not embed owner claims that bypass these controls. Unsupported person-specific claims are withheld for review. A request that absent Alex should do something can become an unassigned “Confirm responsibility for…” follow-up only when that follow-up was actually agreed; otherwise omit it as an action.

Review and publication experience

The meeting page shows “Preparing summary” after the call, then “Draft needs review”. The host is the default reviewer; existing authorised meeting editors may also review. If no reviewer is available, retain the draft without sending owner emails.

Display separate Decisions and Actions sections. Each item links to its transcript passage and recording timestamp. Offer edit, delete and add controls. Actions show task, owner and due date. An attendance-only owner picker explains why invitees or unresolved guests cannot be selected. Provide a visible “Needs owner” state rather than silently dropping useful work.

Reviewers must confirm each chosen owner and can publish with unresolved actions. The publish dialogue shows recipients and unresolved-action count, with an explicit “Publish and email owners” control. Actions without owners remain visible but produce no email. Keyboard navigation, labelled controls and screen-reader status announcements are required.

Published records show their revision and approval time. Subsequent edits create a new revision. Ownership changes require the same validation and explicit publication, then notify affected owners with a correction. Never silently replace a published record with regenerated content.

Engineering contract and failure behaviour

Use meeting occurrence IDs, not recurring calendar series IDs. Store transcript version, attendance evidence version, model/prompt version, summary revision, item IDs, evidence spans, owner ID, approval actor/time and delivery status. Keep internal candidates separate from approved owner fields.

Process states are waiting for transcript, generating, needs review, published and failed. Trigger generation only after transcript finalisation. Transcript changes invalidate unpublished approvals and create a fresh draft; published revisions remain intact until a replacement is approved.

Enforce owner eligibility server-side on draft writes, publication, exports and notification dispatch. Manual edits and direct API calls cannot bypass it. If attendance evidence is missing, contradictory or unavailable, block ownership and sending rather than falling back to calendar data. Record rejection reasons for support without exposing transcript content in routine logs.

Use transactional publication and an outbox. Delivery keys combine meeting, published revision and recipient so retries do not duplicate emails. Email only approved actions owned by that recipient, with a permission-checked meeting link. Resolve addresses from verified identities, never model output. Block delivery if permissions or identity validation fail; show the reviewer the failure. Retry transient failures with backoff for up to 24 hours, then offer manual retry.

Treat transcripts as untrusted data: spoken instructions cannot change permissions, choose recipients or trigger tools. Generation has no email capability. Preserve tenant isolation and apply meeting deletion and retention policies to summaries and queued notifications.

Acceptance scenarios

  1. Invited Pat never joins but is named in an action: Pat is absent from the picker; API assignment is rejected; no owner email is sent.
  1. Uninvited authenticated Lee joins: Lee is eligible and can receive an action after reviewer confirmation and publication.
  1. Several people join through one room device: no individual becomes eligible from its label; actions remain unassigned unless independent authenticated participation exists.
  1. Overlapping speech mislabels a commitment: no owner is preselected; the reviewer can inspect evidence and choose a verified attendee or leave it unassigned.
  1. No decisions or actions are supported: show explicit empty states rather than inventing content. Failed or incomplete transcription shows a retry state and sends nothing.
  1. A forged owner ID, missing attendance record or duplicate publish event cannot create an invalid assignment or duplicate email. Revoked access blocks queued delivery.

Evaluation and release

Build a consented, de-identified evaluation set of at least 200 meetings, deliberately including overlap, shared rooms, absent invitees, uninvited guests, duplicate names, reversals and ambiguous commitments. Human annotators label attendance, final decisions, actions, ownership and evidence; adjudicate disagreements independently. Report each difficult slice separately.

Release gates: zero non-attendee assignments or deliveries across end-to-end and adversarial tests; at least 95% precision and 85% recall for supported decisions and actions, measured separately. Zero observed violations is necessary but insufficient: eligibility enforcement must also pass API and notification-path tests. Model confidence alone never permits assignment.

Pilot with ten opt-in companies before staged rollout. Target draft availability within two minutes of final transcript for 95% of meetings up to 60 minutes, and a 50% reduction in median note-preparation time against pilot baseline. Track owner corrections, unassigned-action rate, publication rate and delivery failures. Any non-attendee assignment disables assignments and sending pending investigation. Expand only after two weeks without a critical violation and acceptable review effort; do not reduce abstention merely to improve coverage.

Grades and run details

Decision model 86 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly76%
  • passAddresses the actual decision91%
  • passRespects explicit constraints58%
  • passIdentifies material uncertainty98%
  • passAvoids unsupported claims74%
  • passProduces the required deliverable88%
  • passSpecifies AI failure behaviour100%
  • passSuccess settles trade-offs95%
  • partialProposes tests that could fail62%
  • partialSays who it's for, and who it isn't66%
  • partialPlans how people will find it88%
Run
Run
#1
Time to output
3.2 min
Submitted
24 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 9

Uses the supplied evidence correctlyWrongMixedRight
Gemini 4 Argon · API

The PRD invents a 10-15 minute note-taking time not in the brief and calls the customer base 'B2B' without evidence.

Gemini 3.8 Flash · API

The output claims that manual note-taking results in lost context, untracked deliverables, and administrative drag, and that speaker labels cannot be trusted for commitments, neither of which appears in the supplied evidence or follows by arithmetic.

GPT-6 Astra · ChatGPT

All factual claims about the current situation are taken directly from the brief or context, with no invented numbers or facts.

Addresses the actual decisionRightWrongRight
Gemini 4 Argon · API

It commits to a human-in-the-loop flow with fallback auto-send after review, explicitly addressing the sales request and constraints.

Gemini 3.8 Flash · API

The output commits to a specific design but does not state what result or condition would change that decision, as required by the criterion.

GPT-6 Astra · ChatGPT

The PRD commits early to a clear answer: generate drafts, host reviews, then publish and email, explicitly rejecting Sales' request for immediate auto-emailing.

Respects explicit constraintsRightMixedRight
Gemini 4 Argon · API

The design builds an Actual_Attendees roster, restricts assignments to that list, and prevents auto-email to non-attendees.

Gemini 3.8 Flash · API

The PRD enforces the hard constraint of never assigning actions to non-attendees via the Eligible Assignee Roster, validation logic, and UI restrictions, and it is written for engineering and design leads within the requested length.

GPT-6 Astra · ChatGPT

The constraint 'never assign an action to someone who was not in the meeting' is enforced through verified attendance, server-side validation, and no preselection from speaker labels.

Identifies material uncertaintyRightMixedRight
Gemini 4 Argon · API

Open questions name latency and diarization confidence availability as unknowns that could change the auto-send timer.

Gemini 3.8 Flash · API

The output does not name specific unknowns that could change the overall approach (e.g., if the 8% error rate is higher in practice) or how they would be resolved.

GPT-6 Astra · ChatGPT

It identifies the key unknown (whether Huddle retains authenticated participant identities and join/leave events) and says how to resolve it and what to do if unavailable.

Avoids unsupported claimsWrongWrongRight
Gemini 4 Argon · API

The 10-15 minute claim is presented as established fact without any support from the supplied context.

Gemini 3.8 Flash · API

It presents interpretations like 'lost context, untracked deliverables, and administrative drag' and 'speaker label alone cannot be trusted' as established facts without labelling them as hypotheses.

GPT-6 Astra · ChatGPT

All interpretations and forecasts are clearly labelled as proposals or decisions; no unsupported claims are presented as established fact.

Success settles trade-offsRightMixedRight
Gemini 4 Argon · API

Success metrics have targets (e.g., 0% false attribution, 60% adoption) and the decision explicitly prioritizes accuracy over speed by using human-in-the-loop.

Gemini 3.8 Flash · API

Success metrics are given with targets but no time frame, and the PRD does not explicitly state which goal wins when two conflict (e.g., speed vs. accuracy), even though the zero-hallucination rule is a hard constraint.

GPT-6 Astra · ChatGPT

Success is defined with measurable targets (draft availability, note-preparation time reduction) and an explicit trade-off rule: do not reduce abstention to improve coverage.

Proposes tests that could failWrongWrongRight
Gemini 4 Argon · API

No proposed test, pilot, or gate with numeric threshold, measurement window, and action to take.

Gemini 3.8 Flash · API

No tests, gates, or kill criteria with numeric thresholds, measurement windows, and specified actions are proposed.

GPT-6 Astra · ChatGPT

Release gates have numeric thresholds (zero non-attendee assignments, 95% precision, 85% recall), a pilot window, and clear actions on failure (disable assignments, investigate).

Says who it's for, and who it isn'tWrongWrongRight
Gemini 4 Argon · API

It does not explicitly name the target users (e.g., meeting hosts) and who is excluded, and no scope decision visibly follows from an exclusion.

Gemini 3.8 Flash · API

The PRD names the target audience (engineering, design, product ops) but does not explicitly name any group it deliberately leaves out, nor does a scope decision visibly follow from such an exclusion.

GPT-6 Astra · ChatGPT

Names primary users (hosts, attendees) and explicitly excludes unverified guests and room-device-only participants, with a scope decision that follows.

Plans how people will find itWrongWrongRight
Gemini 4 Argon · API

There is no rollout plan, no mention of how users discover the feature, and no indication of which teams (support, sales) need to be ready.

Gemini 3.8 Flash · API

The PRD does not describe a staged rollout, how users will discover the feature, or which teams (support, sales) need to be ready.

GPT-6 Astra · ChatGPT

Describes a staged rollout (pilot with ten companies), in-product discovery (meeting page status), and mentions support readiness (rejection reasons for support).

All got right 2

Produces the required deliverableRightRightRight
Gemini 4 Argon · API

It is a PRD for engineering and design leads, within the target length, and provides enough detail to start building.

Gemini 3.8 Flash · API

The output is a complete PRD with specifications, UI mockups, and edge cases, written for engineering and design leads, and falls within the 1,000–1,500 word range.

GPT-6 Astra · ChatGPT

The PRD is complete, written for engineering and design leads, within the 1,000–1,500 word range, and directly actionable.

Specifies AI failure behaviourRightRightRight
Gemini 4 Argon · API

Low-confidence items are flagged for review, assigned to 'Unassigned', and auto-email is paused until the host resolves them.

Gemini 3.8 Flash · API

The PRD specifies fallback to UNASSIGNED on low confidence, deterministic validation that overwrites invalid assignments, a grace period for host review, and handling for missing emails and multi-person rooms.

GPT-6 Astra · ChatGPT

It specifies what happens when uncertain or wrong: block ownership, show retry states, disable assignments on violation, and never fall back to calendar data.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 67% of 81 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT86.984.62None
2Sonnet 5.5withAPI89.279.82None
3GPT-6.1 SolwithAPI84.860.92None
4Opus 5.5withClaude82.860.62None
5GPT-6 LunawithAPI80.759.92None
6Haiku 5.5withAPI76.363.82None
7Gemini 4 ArgonwithAPI69.948.421 capped
8Gemini 3.5 Flash-LitewithGemini52.532.12None
9Gemini 3.8 FlashwithAPI50.428.22None

What you get for the cost

Sonnet 5.5 · API scores within a point of GPT-6 Astra · ChatGPT at a third of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 84, 106× the cheapest (1 of 2 outputs estimated)Sonnet 5.5 · API: task score 83, 35× the cheapestGPT-6.1 Sol · API: task score 72, 14× the cheapestOpus 5.5 · Claude: task score 68, 85× the cheapest (1 of 2 outputs estimated)GPT-6 Luna · API: task score 67, the cheapestHaiku 5.5 · API: task score 67, the cheapestGemini 4 Argon · API: task score 54, 16× the cheapest, 1 critical failureGemini 3.5 Flash-Lite · Gemini: task score 40, 4.4× the cheapest (1 of 2 outputs estimated)Gemini 3.8 Flash · API: task score 39, 9.9× the cheapestGPT-6 Astra · ChatGPTSonnet 5.5 · APIGPT-6.1 Sol · APIOpus 5.5 · ClaudeGPT-6 Luna · APIHaiku 5.5 · APIGemini 4 Argon · APIGemini 3.5 Flash-Lite · GeminiGemini 3.8 Flash · API
Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)At least one critical failure
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API671.0×
Haiku 5.5 · API671.2×
Gemini 3.5 Flash-Lite · Gemini404.4× (partly estimated)
Gemini 3.8 Flash · API399.9×
GPT-6.1 Sol · API7214×
Gemini 4 Argon · API5416×
Sonnet 5.5 · API8335×
Opus 5.5 · Claude6885× (partly estimated)
GPT-6 Astra · ChatGPT84106× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Writing the requirements document a team will build and test against.

Why it matters

A PRD is where ambiguity becomes either a decision or a bug. For AI products it must also say what happens when the model is uncertain or wrong. Most generated PRDs skip that part.

What good looks like

  • States the user problem and the decision the PRD enables
  • Specifies behaviour under uncertainty, failure and refusal
  • Names eval criteria and a launch bar
  • Defines success precisely enough to settle trade-offs
  • Says what is out of scope

Deliberately not measured

  • Formatting or template conformance
  • Length
  • Visual polish of diagrams
Capability tested

Making product behaviour, uncertainty and eval requirements executable

The failure we’re looking for

A generic feature spec that ignores AI failure behaviour

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

AI product PRD (core) · Conventional product PRD