Tasks / Product Craft

Write a PRD

Can the model turn a brief into a spec engineers could build from, including how an AI feature behaves when it is wrong?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 28% were usable with at most a quick edit.

Reliably right

  1. Specifies AI failure behaviour100% pass
    It specifies what happens when uncertain or wrong: block ownership, show retry states, disable assignments on violation, and never fall back to calendar data.
    GPT-6 Astra · ChatGPT · Meeting summaries with action items
  2. Builds the regulator's rules in100% pass
    All three rules (2-business-day acknowledge, 15-business-day resolve or explain, no harder than paying) appear as testable requirements R8, R10, R13.
    Sonnet 5.5 · API · Disputes that aren't disputes
  3. Produces the required deliverable96% pass
    The PRD is complete, written for engineering and design leads, within the 1,000–1,500 word range, and directly actionable.
    GPT-6 Astra · ChatGPT · Meeting summaries with action items

Where it slips

  1. Says who it's for, and who it isn't35% pass
    Does not name the target users (individual account holders) or explicitly exclude a group (e.g., business accounts), missing a required scope decision.
    Sonnet 5.5 · API · Disputes that aren't disputes
  2. Proposes tests that could fail36% pass
    The A/B expansion gate relies on a fraud-reversal guardrail but does not specify a numeric threshold, leaving the test incomplete.
    Sonnet 5.5 · API · Disputes that aren't disputes
  3. Plans how people will find it44% pass
    Gives a staged rollout and discovery method (reviewer notification), but does not name the teams that need to be ready (e.g., support, sales) beyond technical dependencies.
    GPT-6.1 Sol · API · Meeting summaries with action items

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Success settles trade-offs

    Does the PRD define success as a measurable outcome with a target and time frame, and say which goal wins when two of them conflict?

    Passes when Names a success metric with a target and time frame, and gives at least one explicit trade-off rule (for example, accepting lower coverage to keep precision above a stated level).

  2. Says who it's for, and who it isn't

    Does the PRD name the specific users it is for, say explicitly who it is not for, and make at least one scope decision that follows from that choice?

    Passes when Names the target users and at least one group it deliberately leaves out, and a scope decision visibly follows from the exclusion.

  3. Plans how people will find it

    Does the PRD say how the feature reaches people: how it rolls out, how users discover it, and who needs to be ready for it (support, sales), rather than stopping at what to build?

    Passes when Gives a staged rollout, a way users will discover the feature, and the teams that need to be ready, all fitted to this feature.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Proposes tests that could fail · and 1 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

Write a PRD for AI-generated meeting summaries that list the decisions made and assign action items to attendees. The PRD is for the engineering and design leads who will build it: write it so they could start work from it. Aim for 1,000–1,500 words.

What the model was given4 items: Scenario, Known limits, Constraint, Request from Sales
ScenarioHuddle is a video-meeting app used by 3,000 companies. Meetings are recorded and transcribed with speaker labels, and the attendee list comes from the calendar invite. Today people write their own notes after the call.
Known limitsSpeaker labels are wrong about 8% of the time, usually when people talk over each other or several people join from one meeting room. About 15% of meetings include someone who joined without a calendar invite.
ConstraintSummaries must never assign an action to someone who was not in the meeting.
Request from SalesSales wants action items emailed automatically to their owners as soon as each meeting ends.
What a strong answer doesThe answer key the graders mark against

Specifies attribution confidence, editing, and what happens on mis-assignment; keeps a person reviewing before action items go out.

Critical failures (cap the score at 40)
  • Assigns actions to non-attendees
Task

Mid-level · v1.20

The outputs

9 setups answered this brief.

Usable with a quick edit6 of 11 checks passed

The PRD is detailed and usable; a quick edit to remove the unsupported time claim and add a brief rollout note would make it complete.

Output

Product Requirements Document: AI Meeting Summaries & Action Items

Product: Huddle Author: Product Management Target Audience: Engineering & Design Leads Status: Ready for Kickoff

---

1. Overview & Problem Statement

Huddle currently records and transcribes video meetings for 3,000 B2B customers. However, users still spend 10–15 minutes after every call reviewing transcripts or typing up raw notes to document decisions and action items.

We are building an AI-powered post-meeting summary feature that automatically extracts Key Decisions and Action Items (with assigned owners) from meeting transcripts.

The Core Tension: Speed vs. Accuracy

Sales has requested that we automatically email action items to their owners the moment a meeting ends. However, we have three strict operational realities: 1. Hard Constraint: We must never assign an action item to someone who did not attend the meeting. 2. Data Gap 1 (The 15% problem): 15% of meetings include uninvited guests (e.g., forwarded links, conference rooms) who are missing from the calendar invite. 3. Data Gap 2 (The 8% problem): Speaker diarization/labels are wrong 8% of the time, primarily due to crosstalk or multiple people sharing one room mic.

Product Decision: If we auto-send emails immediately after the meeting ends, the 8% speaker attribution error rate and the 15% calendar mismatch rate will inevitably cause us to violate our hard constraint or embarrass users by emailing the wrong people. Therefore, v1 will use a "Human-in-the-Loop" flow with a fallback auto-send. For high-confidence extractions, emails will send automatically after a 30-minute review window. For low-confidence extractions (crosstalk, unverified guests), we will hold the email until the meeting host approves the summary.

---

2. Goals & Success Metrics

MetricTargetRationale
:---:---:---
Summary Adoption Rate> 60% of recorded meetingsMeasures core value of the feature (viewed or shared).
False-Absentee Attribution Rate0.0%Hard Constraint. No non-attendee is ever assigned an item.
Assignee Correction Rate< 10% of assigned action itemsMeasures how well our system handles the 8% diarization error.
Host Review Completion> 50% within 30 minutesValidates whether the host-in-the-loop notification UX works.

---

3. System Architecture & The "Actual Attendee" Roster

Right now, Huddle relies on the calendar invite for the attendee list. To obey our hard constraint, Engineering must build a unified `Actual_Attendees` service that fires when a meeting ends.

Defining an "Actual Attendee"

A person is only eligible to be assigned an action item if they exist in the `Actual_Attendees` array for that specific meeting session. Calendar invitees who did not join the call are explicitly excluded.

  1. Authenticated Huddle Users: Anyone who joins the call while logged into a Huddle account is added to `Actual_Attendees` (solves the 15% uninvited guest problem for logged-in users).
  2. Unauthenticated Web Guests: Guests joining via link must type their display name in the pre-call lobby. They are added to `Actual_Attendees` with `email: null`.
  3. Conference Room Hardware: When a room system joins (e.g., "4th Floor Conf Room"), the room itself is not an assignable person. (See Edge Cases in Section 4).

---

4. Functional Requirements (Engineering)

4.1 LLM Extraction Pipeline

* Trigger: Fires upon completion of the final meeting transcript. * Inputs: 1. Time-stamped transcript with speaker labels. 2. Diarization confidence scores per utterance. 3. The `ActualAttendees` JSON array (Name, UserID, Email, JoinMethod). 4. The `CalendarInviteesNoShow` array (Explicit negative constraint list). * Prompting & Guardrails: * Decisions: Extract 1–7 bullet points summarizing agreed-upon outcomes. Include timestamp citations. * Action Items: Extract tasks, deadlines (if mentioned), and the owner. * Assignment Logic: The LLM may only populate the `owner` field using an exact match from `ActualAttendees`.

4.2 Handling the Constraints & Edge Cases

  • Scenario A: Someone on the call says, "Let's have Sarah do this," when Sarah is NOT in `Actual_Attendees`.
  • System Behavior: The system must not assign the task to Sarah. Instead, it assigns the task to the speaker who volunteered Sarah (or leaves it "Unassigned") and rewrites the task text to: "Follow up with Sarah to [do X]."
  • Scenario B: The 8% Speaker Label Error (Crosstalk / Low Confidence).
  • System Behavior: If an action item is derived from an utterance where someone says "I will do X", the pipeline checks the diarization confidence score for that timestamp block. If confidence is `< 0.85` or overlaps with another speaker within 1.5 seconds, the system sets `owner: Unassigned`, sets `suggestedowner: [SpeakerID]`, and flags the item as `needs_review: true`.
  • Scenario C: Multi-person Conference Rooms.
  • System Behavior: Audio coming from a single hardware node containing multiple voices cannot reliably attribute "I'll take that" to an individual. If the speaker label is a Room Resource or has multi-voice flags, the LLM may only assign an action item if a name is spoken explicitly in the third person (e.g., "Thanks Dave, so Dave will pull the Q3 numbers" AND "Dave" is matched to a Huddle user via calendar invite + confirmation). Otherwise, set `owner: Unassigned` (`needs_review: true`).
  • Scenario D: The 15% Uninvited Guest.
  • System Behavior: If "Alex" joined via a forwarded link without logging in, he is in `ActualAttendees` as `{name: "Alex (Guest)", email: null}`. The system can assign action items to "Alex (Guest)" in the UI, but flags the item as `missingemail: true`.

4.3 The Email Dispatch Engine (Addressing the Sales Request)

To satisfy Sales without sending hallucinated assignments, we will implement a Two-Track Dispatch Engine:

  • Track 1: High-Confidence Auto-Send (30-Minute Delay)
  • Condition: All extracted action items have `owner != Unassigned`, `needsreview == false`, and `missingemail == false`.
  • Behavior: The host gets an immediate Slack/Email ping: "Your meeting summary is ready. Action items will be emailed to owners in 30 minutes unless you edit them." If the host makes no changes within 30 minutes (or clicks "Send Now"), emails are dispatched to action item owners.
  • Track 2: Low-Confidence Hold
  • Condition: One or more action items are flagged `needsreview: true` or `missingemail: true`.
  • Behavior: Auto-send is paused. The host is notified: "Your meeting summary has 2 action items that need your confirmation before we email the team." Emails only trigger once the host resolves the flags and clicks "Approve & Send".

---

5. User Experience & Design Requirements

5.1 Pre-Meeting Lobby (Small tweak for Guest ID)

  • Unauthenticated Guests: Update the pre-join screen. Require both Full Name and Work Email (Optional) before joining. Explaining "Used to send you meeting notes" will convert a high percentage of the 15% uninvited guests into identifiable users.

5.2 Post-Meeting Summary View

The Meeting Host (and participants) will access the summary inside the Huddle web/desktop app under the "Past Meetings" tab.

* Layout: Split screen. Left side: Video player + searchable transcript. Right side: AI Summary widget. * Summary Widget Sections: 1. Status Banner: Shows dispatch state (e.g., "Auto-sending to 4 owners in 24 mins [Send Now] [Pause]" OR "Approval Required before emailing"). 2. Decisions Made: Bulleted list. Hovering over a decision highlights the corresponding timestamp in the transcript and lets the user click to play that 15-second clip. (Crucial for building user trust). 3. Action Items List: Each row contains: * Checkbox (for completion tracking). * Task description (inline editable). * Assignee Dropdown Pill. * Source timestamp link.

5.3 The Assignee Dropdown & "Needs Review" UI

  • Restricted Roster: When a user clicks the Assignee Pill to change or add an owner, the dropdown menu only lists people from the `Actual_Attendees` array.
  • Design detail: Below a divider line at the bottom of the dropdown, show calendar invitees who didn't show up in greyed-out, unclickable text with a tooltip: "Did not attend meeting." (This prevents users from thinking the dropdown is broken while enforcing our constraint—unless someone was hiding in a conference room, see below).
  • Conference Room Override: If a meeting included a Room system, include an option in the dropdown: `+ Add person who was in [Conf Room Name]`. Clicking this allows the host to pick from the org directory, explicitly confirming that the person was physically present in the room.
  • Low-Confidence Visual State: Action items flagged with `needs_review: true` (due to the 8% diarization issue) should have an amber highlight around the Assignee Pill reading: `Suggested: [Name] ?`. Clicking it lets the host confirm with one click.

5.4 The Action Item Email

  • Subject: `Action Items from [Meeting Title] ([Date])`
  • Body:
  • Personalized top section: "Assigned to You (2):" followed by their specific tasks.
  • Context section below: "All Decisions & Team Action Items" so they have full context.
  • Footer CTA: `[View Full Recording & Transcript in Huddle]`.

---

6. Out of Scope for v1

  • 3rd-Party Task Manager Integrations: Pushing tasks to Jira, Asana, Linear, or Salesforce (planned for v1.1 after we validate extraction accuracy).
  • Cross-Meeting Action Item Tracking: A global "My Tasks" dashboard across all meetings. v1 is strictly scoped to the single-meeting view and email dispatch.
  • Voiceprint Enrollment: Having users record their voice to fix the conference room diarization problem. We will rely on UI fallbacks for v1.

---

7. Open Questions for Kickoff

  1. Engineering: What is the p95 latency for transcript generation + LLM summary extraction on a 60-minute call? Sales expects "as soon as the meeting ends"—if our pipeline takes 8 minutes, do we need to adjust the 30-minute auto-send timer?
  2. Design: If a meeting has zero decisions and zero action items (e.g., a casual 1:1 catch-up), what should the empty state look like, and should we suppress the post-meeting notification entirely?
  3. Engineering: Can our current diarization vendor expose word-level confidence scores via their webhook, or do we need to calculate crosstalk purely by checking overlapping speaker timestamps?

Claims the judge couldn’t find in the brief

  • Users still spend 10–15 minutes after every call reviewing transcripts or typing up raw notes
  • Huddle currently records and transcribes video meetings for 3,000 B2B customers

Grades and run details

Decision model 77 · LLM judge 58
Decision model checks
  • partialUses the supplied evidence correctly16%
  • passAddresses the actual decision48%
  • passRespects explicit constraints34%
  • passIdentifies material uncertainty59%
  • partialAvoids unsupported claims20%
  • passProduces the required deliverable83%
  • passSpecifies AI failure behaviour100%
  • passSuccess settles trade-offs69%
  • partialProposes tests that could fail59%
  • partialSays who it's for, and who it isn't61%
  • partialPlans how people will find it34%
Run
Run
#1
API response time
27 s
Submitted
13 Oct 2026

Usable with a quick edit9 of 11 checks passed

The PRD is thorough, evidence is correct, and the missing measurement windows and team readiness can be added with a quick edit without redoing any reasoning.

Output

PRD: AI Meeting Summaries and Action Items

Product: Huddle Audience: Engineering and Design leads Status: Proposed MVP Launch decision: Ship reviewed summaries first; defer immediate, unreviewed owner emails.

1. Problem and opportunity

Huddle serves 3,000 companies. Meetings are recorded and transcribed with speaker labels, but attendees currently write their own notes afterward. This creates duplicated effort, inconsistent records of decisions, and missed follow-through.

We will generate a concise meeting summary containing decisions and action items, with evidence linking each item to the recording.

Two known limitations shape the design:

  • Speaker labels are wrong approximately 8% of the time, especially during overlapping speech and shared-room participation.
  • Approximately 15% of meetings include someone absent from the calendar invite.

Therefore, neither speaker labels nor calendar invitations are sufficient evidence for assigning work.

Non-negotiable requirement: An action must never be assigned to someone who did not attend the meeting. When attendance or identity cannot be established, leave the action unassigned.

2. Goals and non-goals

Goals

  • Reduce time spent producing post-meeting notes.
  • Capture decisions accurately, distinguishing decisions from proposals.
  • Extract actionable commitments with owners and deadlines when supported.
  • Include verified, uninvited attendees without treating all invitees as attendees.
  • Make every decision and action inspectable and correctable.
  • Deliver owner notifications only after safe assignment and publication.

Non-goals for MVP

  • Live summaries or action assignment during meetings.
  • Assigning work to absent colleagues, even if requested during the meeting.
  • Inferring deadlines or assigning owners based on job title.
  • External task-system integrations.
  • Automatically publishing unreviewed summaries.
  • Resolving identity using voice biometrics.

3. Users and primary workflow

Reviewer: Meeting organizer, or an explicitly designated attendee. Reviews and publishes the summary.

Attendee: Reads the summary, follows evidence links, and reports corrections.

Action owner: Receives published actions assigned to them.

Workflow:

  1. Meeting ends; Huddle completes transcript processing.
  2. Huddle reconciles actual attendance and generates a private draft.
  3. Reviewer checks decisions, unresolved ownership, and attendance exceptions.
  4. Reviewer edits and publishes.
  5. Huddle makes the summary available to authorized viewers and, where enabled, emails owners their approved actions.

If no eligible reviewer is available, the draft remains unpublished. A non-attending organizer may designate an attending reviewer but cannot resolve attendance exceptions themselves.

4. Attendance and identity requirements

Engineering must establish an attendance ledger separate from the calendar invite.

Each record contains:

  • Meeting ID and stable person ID, where resolved.
  • Join/leave intervals and participant-session IDs.
  • Identity source: authenticated join, verified guest identity, or shared-room roster.
  • Verification state and evidence.
  • Reviewer confirmation details, where required.

Calendar invitees are suggestions for identity matching, not proof of attendance. A directory search result is also not attendance evidence.

Assignment eligibility

An owner must map to a unique person with verified attendance in that meeting:

  • Authenticated individual join: Eligible when attendance telemetry records the person’s session.
  • Guest join: Eligible only after identity is resolved and verified; a display name alone is insufficient.
  • Shared meeting room: The room endpoint proves room participation, not individual presence. A participating reviewer must explicitly confirm each person physically present before they become eligible.
  • Uninvited attendee: Eligible through the same verification rules; invitation status is irrelevant.
  • Invited but absent person: Ineligible.

If telemetry is unavailable, identity is ambiguous, or attendance is disputed, block assignment. Keep the action unassigned with an explanation.

The owner picker must show only eligible attendees. Every write path—including generation, manual edits, APIs, and publication—must enforce eligibility server-side. Model confidence must never override this rule.

These controls cannot independently observe who is physically in a room. If trustworthy roster confirmation is unavailable, shared-room individuals remain ineligible rather than weakening the constraint.

5. Summary and extraction requirements

The draft has four sections:

  1. Overview: Two to four sentences describing the meeting’s purpose and outcome.
  2. Decisions: Clear statements of agreed outcomes.
  3. Action items: Task, owner or “Unassigned,” deadline if explicit, and status.
  4. Needs review: Uncertain decisions, ownership, identity, or attendance.

Each decision and action must include transcript-span references and recording timestamps. Preserve the original supporting text even when the reviewer edits the final wording.

Decisions

Capture explicit agreement or a clearly accepted conclusion. Do not convert brainstorming, recommendations, or unresolved debate into decisions. If discussion reverses an earlier decision, represent the final outcome and retain supporting evidence.

Actions

Extract a concrete deliverable or next step. Record a deadline only when explicitly stated; normalize relative dates using the meeting’s date and timezone.

Examples:

  • “I’ll send the revised deck tomorrow.” → Task with a candidate owner, subject to attribution review.
  • “Someone should update the deck.” → Unassigned action.
  • “Ask Priya to update it,” when Priya was absent → Unassigned action; note that an absent person was mentioned.
  • “We could revisit this next quarter.” → Not an action without a clear commitment.

Speaker labels provide evidence, not authority. Overlap, shared-room speech, ambiguous pronouns, and conflicting commitments must trigger ownership review. Candidate owners are draft suggestions, not assignments, until approved.

6. Review experience

The summary page opens in Draft state with a clear “Review before sharing” banner.

Design must provide:

  • Editable decision and action cards.
  • Evidence links opening the relevant transcript and recording segment.
  • Visible distinction between verified attendance and uncertain attribution.
  • “Unassigned” as a first-class owner option.
  • A dedicated queue for identity and attendance exceptions.
  • A publish preview showing recipients and notification settings.

For each assigned action, the reviewer must approve the owner. Bulk approval is permitted for straightforward items, but flagged attribution requires item-level review.

Attendance verification and ownership approval are separate controls: confirming that someone attended does not prove they accepted a task.

Publication is blocked if any assigned owner is ineligible. Unassigned actions and unresolved decision candidates do not block publication; uncertain decisions must be omitted from the final Decisions section or explicitly labeled unresolved.

7. Email policy and Sales request

Sales requests automatic emails immediately after each meeting. MVP will not send unreviewed action assignments. Given the known speaker errors, immediate delivery would spread incorrect ownership before correction.

Instead:

  • Notify the reviewer when the draft is ready.
  • Default owner emails to off; enable through workspace settings.
  • When enabled, automatically email approved owners after publication.
  • Send each owner only their actions, meeting title, deadlines, and an authorized summary link.
  • Never email a calendar invitee solely because they were invited.
  • Do not send to guests without a verified address and permitted access.

Email dispatch must recheck eligibility and publication version. Use idempotency keys to prevent duplicate sends. Published ownership changes generate a correction for the previous recipient and an updated notification for the new eligible owner.

Future immediate automation requires a separate launch decision and evidence that it preserves the attendance constraint.

8. Technical implementation

Use an asynchronous pipeline:

`Meeting ended → Transcript ready → Attendance reconciled → Draft generated → Reviewed → Published → Notifications`

Proposed records:

  • `Summary`: meeting ID, transcript version, state, generated content, revision.
  • `Decision`: text, evidence spans, review state.
  • `Action`: task, proposed owner, approved owner, deadline, evidence, review state.
  • `AttendanceRecord`: identity, evidence, verification state.
  • `AuditEvent`: actor, change, timestamp, revision.

Generation returns schema-validated structured data. Treat transcript content as untrusted input: instructions spoken during a meeting must not alter system rules, access controls, or recipient selection.

Owner writes and publication run through a centralized eligibility validator. Notification jobs check the current revision before sending. Attendance corrections invalidate affected assignments and cancel queued notifications. Maintain an audit trail rather than silently rewriting history.

9. Reliability, privacy, and access

Target draft availability: within five minutes of transcript completion for 95% of supported meetings.

Generation failures must leave recordings and transcripts usable, show retry status, and send no owner emails.

Drafts are visible only to reviewers. Published summaries inherit meeting-content access rules; attendance alone does not grant access. Recording evidence links must enforce existing permissions. Apply workspace retention and deletion policies to summaries, attendance evidence, and generated artifacts.

10. Success metrics and acceptance

Proposed launch gates

  • Zero assignments to ineligible attendees across automated invariant tests.
  • Zero emails to ineligible owners in the pilot; any confirmed incident pauses rollout.
  • At least 90% of published decisions require no substantive reviewer correction.
  • Median review time below three minutes for meetings under one hour.
  • At least 30% less self-reported note-taking time in the pilot.

Measure owner-correction rate separately from attendance violations: a wrong attending owner is still a quality failure.

Required acceptance cases

Test absent invitees, verified uninvited guests, shared rooms, overlapping speakers, duplicate names, failed identity resolution, missing telemetry, absent people named in requests, transcript reprocessing, attendance revoked before dispatch, and repeated notification retries.

In every uncertain-attendance case, expected behavior is an unassigned action and no owner email.

11. Rollout and dependencies

Start with internal meetings, then an opt-in pilot across varied company sizes and shared-room usage. Expand only after launch gates pass.

Before implementation, confirm availability of participant-session telemetry, guest identity verification, room-roster confirmation, reviewer permissions, and verified email addresses. If these dependencies are missing, ship summaries with unassigned actions for affected participants rather than substituting calendar membership as attendance proof.

Grades and run details

Decision model 86 · LLM judge 83
Decision model checks
  • passUses the supplied evidence correctly70%
  • passAddresses the actual decision83%
  • passRespects explicit constraints62%
  • passIdentifies material uncertainty52%
  • passAvoids unsupported claims62%
  • passProduces the required deliverable88%
  • passSpecifies AI failure behaviour100%
  • passSuccess settles trade-offs74%
  • partialProposes tests that could fail82%
  • partialSays who it's for, and who it isn't77%
  • partialPlans how people will find it83%
Run
Run
#1
API response time
47 s
Submitted
29 Sept 2026

Needs more than a quick edit9 of 11 checks passed

The PRD assumes the existence of session join/leave logs and room device behavior that are not in the supplied context, so a PM would need to verify those before proceeding.

Output

PRD: AI Meeting Summaries with Decisions and Action Items

Status: Draft for engineering and design leads Product: Huddle

1. Problem

Huddle records and transcribes meetings with speaker labels, but attendees still write their own notes afterward. Notes are late, inconsistent, or never written, so decisions get lost and action items have no clear owner. We will generate a summary after each meeting that lists decisions and proposes action items with owners.

2. Goals and non-goals

Goals - Produce a summary within 5 minutes of meeting end: decisions, action items (task, owner, due date if stated), and open questions. - Never assign an action item to someone who was not in the meeting (hard constraint, see §4). - Keep the host's review to under 60 seconds for a typical meeting.

Non-goals (v1) - Fully automatic emailing of action items to owners (see §3). - Cross-meeting task tracking or integrations with Jira, Asana, and similar tools. - Summaries for meetings without a transcript.

3. Decision on the Sales request

Sales asked for action items to be emailed automatically to owners the moment a meeting ends. We are not building that for v1. Instead, we will send the draft summary to the host at meeting end. The host can send it to owners with one click, and owners are emailed on send.

Reasons: - Speaker labels are wrong about 8% of the time. An automatic email would sometimes tell a real person they own a task they never agreed to. Because it goes out with no human check, the error becomes visible to a customer's colleagues. - Email is irreversible, so a wrong or non-attendee assignment cannot be quietly fixed afterward. - The underlying Sales need is that owners hear about their tasks quickly and nothing falls through the cracks. A host prompt at meeting end with one-click send meets that need at the cost of a few seconds.

Path to automation: Auto-send will be an opt-in, per-workspace setting, considered after launch once the gates in §9 are met. It would send only high-confidence, presence-verified items and leave everything else for host review. Engineering should build the send pipeline so that auto-send is later a policy change rather than a rebuild.

4. The hard constraint: who can own an action item

Rule: An action item's owner must be a participant who was actually in the meeting when the item was discussed. Anyone else is ineligible.

This rule cannot rely on the calendar invite, and it cannot rely on the model behaving well.

Attendance source. The eligible-owner roster is built from Huddle's session join/leave logs, not from the calendar invite. This handles both failure modes: - Invitees who never joined are not on the roster. - The roughly 15% of meetings with uninvited joiners include those people, matched to a user account when possible.

Presence at time of item. A person is eligible for an item only if they were in the session at the timestamp of the relevant utterance. Someone who left at minute 10 cannot own an item from minute 40.

Enforcement is structural, not prompt-based. 1. The extraction step receives the roster as a list of participant IDs and must return an owner ID from that list or `null`. 2. A deterministic validator runs after the model. It rejects any owner ID not on the roster or not present at the item's timestamp, and it converts that item to unassigned. 3. The UI and email layers accept only validated owner IDs. There is no free-text owner field on the model output path. 4. Names mentioned in the transcript who were not present ("have Dana look at it") produce an unassigned item. The text may keep the speaker's wording, but Dana gets no assignment and no email.

Shared rooms. A room device usually appears as one participant, so individuals in the room are not in the presence log. We cannot verify that a named person is in the room. Such items become unassigned by default, and the host can attest who was in the room during review (§6). That attestation counts as human-verified presence.

Uninvited joiners without an account (guests) can be eligible owners, shown by display name. They receive email only if the host supplies an address during review. We never guess addresses.

5. Handling speaker-label errors

The 8% label error rate is concentrated in crosstalk and shared rooms. Eligible-owner enforcement does not solve it, because a mislabel can still pick the wrong person who was present. Mitigations:

SignalHandling
Owner named explicitly ("Priya, can you send the deck?") and Priya is present, followed by assent or no objectionHigh confidence. Name-based assignment does not depend on speaker labels.
Self-assignment ("I'll send it")Depends on the label. High only if the segment is not in a crosstalk or shared-room span and label confidence is above threshold. Otherwise Medium.
Assignment inferred from context, or the segment overlaps other speakersLow. Default to unassigned with a suggested owner the host can accept.
Speaker is a shared-room deviceNever treated as a person. Unassigned unless a named individual is explicitly addressed and verified.

Confidence tiers (High/Medium/Low) are stored on each item, shown to the host in review, and used later to gate auto-send. Thresholds are set empirically during the evaluation in §9. The tiers themselves are a v1 requirement.

6. User experience

Host flow 1. The meeting ends and the host gets a notification: "Summary ready to review." 2. The review screen lists: - Decisions as short statements, each linked to the transcript moment. - Action items with the task, owner chip, due date if stated, and a confidence indicator. - Unassigned items grouped at the top of the actions list, so gaps are visible. 3. The host can edit text, reassign from a picker limited to eligible people, delete items, or add room attendees ("Who was in the room?"). Adding room attendees makes them eligible for items in the room segments. 4. Send emails each owner their own items plus the decisions. Unassigned items go to no one.

Design requirements - Every item has a "jump to source" link that plays the transcript segment. Reviewing should be verification, not re-listening. - The owner picker never offers non-attendees. If the host wants to assign to an absent person, show a separate option, "Send as a follow-up (not from the meeting)," which is a different item type with clearly different wording and no "you agreed to" language. Design to decide whether this ships in v1; the default is that it does not. - Low-confidence owners appear as suggestions ("Suggested: Sam") and never as assignments. - Email copy says "Action item from Meeting name, date", includes the source excerpt, and offers a "This isn't mine" link that notifies the host. This gives owners a correction path. - If the host does not review, send a reminder at 24 hours. Nothing is sent to owners without host action in v1.

7. Functional requirements

  1. Roster service: builds the per-meeting roster with join/leave intervals from session logs and links to user accounts where possible.
  2. Extraction: from the transcript and roster, produce decisions, action items, and unassigned items. Each action item has task text, owner ID or null, due date or null, source timestamps, and a confidence tier.
  3. Validator: enforces §4 as code with unit tests, independent of the model, and logs every rejection.
  4. Review UI: as described in §6, on web and desktop. Mobile is view-only for v1.
  5. Delivery: per-owner emails on host send, with a record of who was emailed. Sends are idempotent, and edits after sending trigger an explicit "send update" action.
  6. Audit log: stores for each item the model output, validator result, host edits, and send events.
  7. Admin setting: workspace-level toggle for summaries, plus retention aligned with existing transcript retention.

8. Edge cases to specify and test

  • Invitee never joined but is mentioned as owner.
  • Uninvited joiner takes an action; the guest has no account.
  • Person leaves and rejoins (multiple intervals).
  • Two people share a first name; ownership must resolve by roster, and ambiguity means unassigned.
  • Whole meeting in a shared room (all items default unassigned until the host attests).
  • Multiple languages, or a transcript with large gaps.
  • Nobody commits to anything: show "No action items detected" rather than inventing some.

9. Success metrics and launch gates

Launch gates (must all pass) - Zero non-attendee assignments across a labeled evaluation set of at least 500 meetings, deliberately oversampled for uninvited joiners, shared rooms, no-show invitees, and crosstalk. Any production occurrence afterward is a Sev-1 incident. - Among High-confidence items, wrong-attendee assignment rate of 3% or less (target, to be validated against the evaluation set; propose a stricter figure if data supports it). - Median summary delivery within 5 minutes of meeting end.

Post-launch metrics - Share of meetings where the host sends the summary within 1 hour. - Host edit rate per item, and owner "This isn't mine" rate. - Share of action items left unassigned. Some is healthy, but a high rate means the model is too timid.

Auto-send criteria (future): two consecutive months of High-tier precision above the agreed bar and no non-attendee incidents. Sales and Support are consulted before enabling.

10. Risks and open questions

  • Sales disagreement. Sales may see the host step as friction. We should share §3 with them early and offer a pilot of auto-send with a few design-partner accounts after the gates are met.
  • Hosts skipping review. If sending rates are low, the fix is better reminders and faster review, not removing the check.
  • Presence data quality. Do our session logs reliably capture join/leave for all clients and dial-in users? Engineering to confirm in week 1.
  • Guest privacy. Can we email or display guest names without consent, given customer policies? Legal to advise.
  • Room attestation. Is host attestation adequate for the constraint, or should room participants confirm individually? Recommend host attestation for v1.
  • Follow-up items for absent people. Ship in v1 or defer? Design and Product to decide.

11. Milestones

  1. Weeks 1–2: roster service, presence data audit, evaluation set labeling begins.
  2. Weeks 3–5: extraction, validator, confidence tiers, offline evaluation.
  3. Weeks 4–7: review UI and email delivery, in parallel.
  4. Weeks 8–9: internal dogfood, gate review.
  5. Week 10: limited beta with design-partner customers, including Sales' accounts.

Claims the judge couldn’t find in the brief

  • Notes are late, inconsistent, or never written.
  • Huddle has session join/leave logs that can build a per-meeting roster with join/leave intervals.
  • A room device usually appears as one participant, so individuals in the room are not in the presence log.
  • Uninvited joiners without an account (guests) can be eligible owners, shown by display name.

Grades and run details

Decision model 91 · LLM judge 75
Decision model checks
  • passUses the supplied evidence correctly49%
  • passAddresses the actual decision95%
  • passRespects explicit constraints56%
  • passIdentifies material uncertainty97%
  • passAvoids unsupported claims52%
  • passProduces the required deliverable86%
  • passSpecifies AI failure behaviour100%
  • passSuccess settles trade-offs97%
  • partialProposes tests that could fail66%
  • partialSays who it's for, and who it isn't67%
  • passPlans how people will find it76%
Run
Run
#1
API response time
49 s
Submitted
29 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 5

Uses the supplied evidence correctlyWrongRightMixed
Gemini 4 Argon · API

The PRD invents a 10-15 minute note-taking time not in the brief and calls the customer base 'B2B' without evidence.

GPT-6.1 Sol · API

All facts about the current situation are taken directly from the brief, with no invented numbers or systems.

Sonnet 5.5 · API

The output invents facts about session join/leave logs, room device behavior, and note quality that are not in the supplied context.

Avoids unsupported claimsWrongRightMixed
Gemini 4 Argon · API

The 10-15 minute claim is presented as established fact without any support from the supplied context.

GPT-6.1 Sol · API

Forecasts like 'immediate delivery would spread incorrect ownership' are clearly presented as reasoning, not as established facts.

Sonnet 5.5 · API

It presents session join/leave logs, room device behavior, and note quality issues as established facts without evidence.

Proposes tests that could failWrongWrongRight
Gemini 4 Argon · API

No proposed test, pilot, or gate with numeric threshold, measurement window, and action to take.

GPT-6.1 Sol · API

Launch gates have numeric thresholds and actions, but lack explicit measurement windows (e.g., pilot duration, number of meetings) needed to read out results.

Sonnet 5.5 · API

Launch gates have numeric thresholds (zero non-attendee in 500 meetings, 3% error rate, 5-min delivery) with clear actions (Sev-1 incident, auto-send criteria).

Says who it's for, and who it isn'tWrongRightRight
Gemini 4 Argon · API

It does not explicitly name the target users (e.g., meeting hosts) and who is excluded, and no scope decision visibly follows from an exclusion.

GPT-6.1 Sol · API

Names target users (reviewer, attendee, action owner) and explicitly excludes absent colleagues and unverified guests, with scope decisions following from those exclusions.

Sonnet 5.5 · API

It names engineering and design leads as readers, excludes non-transcript meetings and mobile editing, and scopes out Jira/Asana integrations.

Plans how people will find itWrongWrongRight
Gemini 4 Argon · API

There is no rollout plan, no mention of how users discover the feature, and no indication of which teams (support, sales) need to be ready.

GPT-6.1 Sol · API

Gives a staged rollout and discovery method (reviewer notification), but does not name the teams that need to be ready (e.g., support, sales) beyond technical dependencies.

Sonnet 5.5 · API

It includes a staged rollout (internal dogfood, limited beta), discovery via host notification, and readiness for Sales and Support.

All got right 6

Addresses the actual decisionRightRightRight
Gemini 4 Argon · API

It commits to a human-in-the-loop flow with fallback auto-send after review, explicitly addressing the sales request and constraints.

GPT-6.1 Sol · API

Commits to a clear launch decision (ship reviewed summaries, defer unreviewed emails) and states what would change it (evidence that immediate automation preserves the attendance constraint).

Sonnet 5.5 · API

The output clearly commits to not auto-emailing action items in v1, gives reasons, and states what would change the decision (future gates).

Respects explicit constraintsRightRightRight
Gemini 4 Argon · API

The design builds an Actual_Attendees roster, restricts assignments to that list, and prevents auto-email to non-attendees.

GPT-6.1 Sol · API

Respects the non-assignment constraint through an attendance ledger, eligibility rules, and server-side enforcement; also meets the PRD form, audience, and word count.

Sonnet 5.5 · API

The hard constraint is enforced structurally via roster, validator, and UI restrictions, and the PRD is written for engineering/design leads within the word limit.

Identifies material uncertaintyRightRightRight
Gemini 4 Argon · API

Open questions name latency and diarization confidence availability as unknowns that could change the auto-send timer.

GPT-6.1 Sol · API

Names specific unknowns (speaker errors, shared-room identity, missing telemetry) and says how they would be resolved (e.g., ship with unassigned actions if dependencies missing, separate launch decision for immediate automation).

Sonnet 5.5 · API

The output lists specific unknowns (presence data quality, guest privacy, room attestation) and how they will be resolved.

Produces the required deliverableRightRightRight
Gemini 4 Argon · API

It is a PRD for engineering and design leads, within the target length, and provides enough detail to start building.

GPT-6.1 Sol · API

Delivers a complete PRD for engineering and design leads, within the requested length, with enough detail to start work.

Sonnet 5.5 · API

The output is a complete PRD with problem, goals, UX, functional requirements, and milestones, usable by engineering and design leads.

Specifies AI failure behaviourRightRightRight
Gemini 4 Argon · API

Low-confidence items are flagged for review, assigned to 'Unassigned', and auto-email is paused until the host resolves them.

GPT-6.1 Sol · API

Specifies that uncertain attendance leads to unassigned actions, no owner emails, and generation failures leave recordings usable with retry status.

Sonnet 5.5 · API

It specifies confidence tiers, unassigned defaults, validator rejections, and a 'This isn't mine' correction path for owners.

Success settles trade-offsRightRightRight
Gemini 4 Argon · API

Success metrics have targets (e.g., 0% false attribution, 60% adoption) and the decision explicitly prioritizes accuracy over speed by using human-in-the-loop.

GPT-6.1 Sol · API

Defines success metrics with targets (e.g., 90% decision accuracy, <3 min review time) and gives an explicit trade-off rule: never assign if uncertain, even if it means unassigned actions.

Sonnet 5.5 · API

It defines success metrics with targets (zero non-attendee assignments, 3% wrong-attendee rate, 5-min delivery) and trade-off rules (precision over coverage, host review over automation).

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 67% of 81 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT86.984.62None
2Sonnet 5.5withAPI89.279.82None
3GPT-6.1 SolwithAPI84.860.92None
4Opus 5.5withClaude82.860.62None
5GPT-6 LunawithAPI80.759.92None
6Haiku 5.5withAPI76.363.82None
7Gemini 4 ArgonwithAPI69.948.421 capped
8Gemini 3.5 Flash-LitewithGemini52.532.12None
9Gemini 3.8 FlashwithAPI50.428.22None

What you get for the cost

Sonnet 5.5 · API scores within a point of GPT-6 Astra · ChatGPT at a third of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 84, 106× the cheapest (1 of 2 outputs estimated)Sonnet 5.5 · API: task score 83, 35× the cheapestGPT-6.1 Sol · API: task score 72, 14× the cheapestOpus 5.5 · Claude: task score 68, 85× the cheapest (1 of 2 outputs estimated)GPT-6 Luna · API: task score 67, the cheapestHaiku 5.5 · API: task score 67, the cheapestGemini 4 Argon · API: task score 54, 16× the cheapest, 1 critical failureGemini 3.5 Flash-Lite · Gemini: task score 40, 4.4× the cheapest (1 of 2 outputs estimated)Gemini 3.8 Flash · API: task score 39, 9.9× the cheapestGPT-6 Astra · ChatGPTSonnet 5.5 · APIGPT-6.1 Sol · APIOpus 5.5 · ClaudeGPT-6 Luna · APIHaiku 5.5 · APIGemini 4 Argon · APIGemini 3.5 Flash-Lite · GeminiGemini 3.8 Flash · API
Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)At least one critical failure
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API671.0×
Haiku 5.5 · API671.2×
Gemini 3.5 Flash-Lite · Gemini404.4× (partly estimated)
Gemini 3.8 Flash · API399.9×
GPT-6.1 Sol · API7214×
Gemini 4 Argon · API5416×
Sonnet 5.5 · API8335×
Opus 5.5 · Claude6885× (partly estimated)
GPT-6 Astra · ChatGPT84106× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Writing the requirements document a team will build and test against.

Why it matters

A PRD is where ambiguity becomes either a decision or a bug. For AI products it must also say what happens when the model is uncertain or wrong. Most generated PRDs skip that part.

What good looks like

  • States the user problem and the decision the PRD enables
  • Specifies behaviour under uncertainty, failure and refusal
  • Names eval criteria and a launch bar
  • Defines success precisely enough to settle trade-offs
  • Says what is out of scope

Deliberately not measured

  • Formatting or template conformance
  • Length
  • Visual polish of diagrams
Capability tested

Making product behaviour, uncertainty and eval requirements executable

The failure we’re looking for

A generic feature spec that ignores AI failure behaviour

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

AI product PRD (core) · Conventional product PRD