Add Speaker Diarization (Speaker Turn Attribution) to SpeechRecognition #226

Summary

We propose extending SpeechRecognition, SpeechRecognitionOptions, and SpeechRecognitionResult to support opt-in speaker diarization—splitting recognition results at speaker turn boundaries (speakerChanged) and attributing each SpeechRecognitionResult to a session-ephemeral relative speaker label (speaker: "1", "2", ...).

(Companion explainer: explainers/speaker-diarization.md; also relates to #105, #192, #208, #213, and #216.)

Motivation

While recent updates to SpeechRecognition enable on-device processing (processLocally: true), MediaStreamTrack and AudioData (#216) inputs, and result timestamps (speechStartTime / speechEndTime, #192), the API still returns one undifferentiated stream of text. Applications cannot tell where one speaker stopped and another began, or which utterances share a speaker:

1. Shared-Mic Conferencing & Interviews: Multiple participants speaking into a shared room microphone collapse into an unattributed wall of text, degrading live captions and meeting summaries.
2. Accessible Multi-Speaker Subtitling (WebVTT <v> Voice Spans & >> Markers): Captioning guidelines require marking speaker changes (>>) and identifying speakers (<v Speaker 1>), which currently cannot be automated when building VTTCues from SpeechRecognitionResult.
3. Eliminating Duplicate In-Page Audio & Embedding Pipelines: Privacy-sensitive apps using processLocally: true (clinical, legal, educational, enterprise) currently must capture raw PCM via getUserMedia(), download separate tens-of-MB speaker models per origin in WebAssembly, compute voice embeddings in page script, and manually reconcile misaligned timestamps against SpeechRecognitionResult.
4. Talk-Time Analytics & On-Device LLM Pipelines: Combining measured speechEndTime - speechStartTime (Update SpeechRecognitionResult with speech timing attributes #192) with anonymous "1", "2" speaker labels enables talk-time analytics and lets apps pass plain-text attributed transcripts to on-device LLMs (e.g., the Prompt API) for role inference ("clinician" vs. "patient") without exposing voice data to either script or the LLM.

Proposed Solution (Web IDL)

enum SpeechRecognitionDiarization {
    "none",
    "speaker-turns",
    "speaker-labels"
};

partial interface SpeechRecognition {
    attribute SpeechRecognitionDiarization diarization;
    attribute unsigned long minSpeakers;
    attribute unsigned long maxSpeakers;
};

partial dictionary SpeechRecognitionOptions {
    SpeechRecognitionDiarization diarization = "none";
};

partial interface SpeechRecognitionResult {
    readonly attribute boolean speakerChanged;
    readonly attribute DOMString? speaker;
};

// Added to SpeechRecognitionErrorCode:
//   "diarization-not-supported"

Example Usage

const recognition = new SpeechRecognition();
recognition.lang = 'en-US';
recognition.continuous = true;
recognition.processLocally = true;
recognition.diarization = 'speaker-labels';
recognition.maxSpeakers = 4;

recognition.onresult = (event) => {
  for (let i = event.resultIndex; i < event.results.length; ++i) {
    const result = event.results[i];
    if (!result.isFinal) continue;
    const text = result[0].transcript.trim();
    const vtt = result.speaker ? `<v Speaker ${result.speaker}>${text}` : text;
    captionTrack.addCue(new VTTCue(result.speechStartTime, result.speechEndTime, vtt));
  }
};

Proposed Algorithm & Edge-Case Rules

1. Configuration & Explicit Failure:
   • diarization, minSpeakers, and maxSpeakers (0 = no hint) are read when start() is called; minSpeakers / maxSpeakers are ignored unless diarization is "speaker-labels".
   • If minSpeakers > 0, maxSpeakers > 0, and minSpeakers > maxSpeakers, start() throws a RangeError.
   • If the user agent cannot provide the requested diarization mode for the active language and processLocally setting, the start session algorithm queues an error event with "diarization-not-supported" (MUST NOT silently downgrade to "none").
2. Result Splitting & speakerChanged Invariant:
   • When diarization is not "none", the user agent ends the current SpeechRecognitionResult at each detected speaker turn (in addition to normal pause endpointing and, when combined with What happens if the language isn't specified when using SpeechRecognition, and how to recognized multiple languages #208, detected language switches).
   • speakerChanged is false for the first result of a session (index === 0) and whenever a new result starts after a pause or language switch by the same speaker; it is true only when the speaker differs from the preceding result. In "speaker-labels" mode, whenever consecutive results both have non-null speaker, speakerChanged equals speaker !== previous.speaker.
3. Label Format, null Semantics & Cardinality:
   • In "speaker-labels" mode, labels are canonical 1-indexed decimal strings ("1", "2", ...) assigned in order of first appearance in the session and reset to "1" on each new start() session.
   • speaker is always null in "none" and "speaker-turns" modes. In "speaker-labels" mode, speaker MAY be null on early interim results (before ~1–2s of a new voice has been heard) or on final results too brief/ambiguous to attribute (e.g., short backchannels like "mm-hmm").
   • If maxSpeakers > 0, the user agent MUST NOT emit more than maxSpeakers distinct labels; minSpeakers acts as an advisory hint.
4. Overlapping Speech, Multi-Channel & Single-Shot (continuous = false):
   • Overlapping speech is attributed to the dominant speaker in v1 (no source separation).
   • Multi-channel separation is out of scope (applications with per-participant channels can split channels via Web Audio and run separate sessions).
   • When continuous is false, the session's single result ends at the first detected speaker turn.
5. Streaming vs. Faster-Than-Realtime (AudioData, Add explainer for speech recognition on AudioData #216) Stability:
   • On isFinal === false interim results, speaker and speakerChanged are provisional; once isFinal === true, they are immutable.
   • For live streams (microphone / MediaStreamTrack), user agents SHOULD balance finalization delay against attribution stability. For AudioData input (Add explainer for speech recognition on AudioData #216), user agents SHOULD leverage queued lookahead (up to end-of-input) so finalized labels are globally consistent across the recording.

Privacy & Security Considerations

• Strictly Ephemeral, Non-Biometric State: Labels are computed solely from the current session's audio. User agents MUST NOT expose raw speaker embeddings, MUST NOT match voices against stored OS/browser/cloud profiles (extending the existing prohibition on personalizing MediaStreamTrack recognition), and MUST discard all voice representations when the session ends, is aborted, or the document unloads.
• Net Data Minimization: Keeping voice embeddings inside the user agent and surfacing only session-scoped "1", "2" labels avoids forcing web pages to capture raw microphone audio (getUserMedia()) and compute retainable voice embeddings in page script.
• Anti-Fingerprinting: Canonical "1", "2" strings prevent engine-prefix leaks; available() reuses SpeechRecognitionAvailabilityStatus with storage-key download masking (Add privacy and security mitigations to the Web Speech API #165); turn bounds add no timing side channel beyond Update SpeechRecognitionResult with speech timing attributes #192.
• Bystander Privacy & Permissions Policy: Session-ephemeral clustering prevents cross-session tracking of secondary speakers/bystanders, and diarization inherits existing "microphone" and "on-device-speech-recognition" ('self', #165) policy gating ("service-not-allowed" / "not-allowed").

Alternatives Considered

• speaker on SpeechRecognitionAlternative: Acoustic diarization describes the audio span rather than individual N-best lexical alternatives; SpeechRecognitionResult matches speechStartTime / speechEndTime (Update SpeechRecognitionResult with speech timing attributes #192) and lang (What happens if the language isn't specified when using SpeechRecognition, and how to recognized multiple languages #208).
• Boolean diarization without "speaker-turns": "speaker-turns" serves live captioning (>>), uses less compute, and avoids computing voice embeddings altogether.
• Implying diarization from quality: "conversation" (Add quality level hint to SpeechRecognitionOptions #186): Quality tiers describe acoustic robustness, whereas diarization alters segmentation and has distinct privacy and model-download requirements.
• Transient onspeakerchange DOM events only: Decouples speaker attribution from cumulative event.results.

Open Questions for WG Discussion

1. Unified Result & Sub-Result Metadata (with Multilingual lang, #208):

   • Should speaker attribution and per-result lang (What happens if the language isn't specified when using SpeechRecognition, and how to recognized multiple languages #208) share the exact same result-splitting rule (ending a SpeechRecognitionResult whenever either speaker or language changes, with speakerChanged === false across same-speaker code-switches)?
   • Is result-level splitting sufficient for v1, or does fragmentation from mid-sentence code-switching / rapid interjections require moving both features together to a shared sub-result segments structure (time ranges on SpeechRecognitionResult vs. word spans on SpeechRecognitionAlternative)?

2. Streaming (interimResults) vs. Offline (AudioData, #216) Label Revisions:

   • Is keeping isFinal === true results strictly immutable sufficient for v1 (using provisional interim updates + slight finalization delay for live streams, and full lookahead clustering for AudioData recordings)?
   • If live streaming engines need post-finalization cluster merges later, which mechanism is preferable:
     • (a) None in v1 (immutable isFinal, current proposal);
     • (b) Non-mutating merge event (speakerlabelsmerged with { from: "3", to: "1" }) that lets apps merge earlier speaker groups without mutating finalized SpeechRecognitionResult objects; or
     • (c) onspeakerlabelchange (updatedResultIndices) updating finalized results in place?

3. Audio Timeline Alignment (#192, #213, #216) & Media Pause/Seek:

   • Turn-split [speechStartTime, speechEndTime] bounds use the session audio timeline (0.0s at first consumed frame in Update SpeechRecognitionResult with speech timing attributes #192 / The timestamp of various events is under-specified, and not implemented per spec in shipped implementations #213, or AudioData.timestamp in Add explainer for speech recognition on AudioData #216).
   • Per The timestamp of various events is under-specified, and not implemented per spec in shipped implementations #213, pausing or seeking an HTMLMediaElement captured via captureStream() requires starting a new SpeechRecognition session—which resets session-scoped speaker labels back to "1". Is this session-reset boundary acceptable for media-element captioning (given AudioData Add explainer for speech recognition on AudioData #216 handles full-file transcription without pause/seek resets)?

4. Privacy Gating, processLocally, and Permissions-Policy:

   • Should "speaker-labels" require processLocally = true (strongest privacy guarantee), or also allow cloud-backed SpeechRecognition implementations that support ephemeral diarization?
   • Should "speaker-labels" be gated by a dedicated Permissions-Policy feature (e.g., "speaker-diarization", default allowlist 'self') and/or distinct error code ("diarization-not-allowed" vs. reusing "service-not-allowed")?

5. Naming & Constraints:

   • Naming bikeshed: diarization vs. speakerDiarization vs. speakerAttribution; speaker vs. speakerLabel; speakerChanged vs. speakerChange.
   • Is treating maxSpeakers as a hard cap and minSpeakers as a soft hint interoperable across engines?

Prior Art & Multi-Engine On-Device Precedent

Published On-Device Diarization Precedent:

• Turn-to-Diarize: Online Speaker Diarization Constrained by Transformer Transducer Speaker Turn Detection (arXiv:2109.11641) — inline transducer turn tokens ("speaker-turns") paired with compact turn embeddings ("speaker-labels").
• Highly Efficient Real-Time Streaming and Fully On-Device Speaker Diarization with Multi-Stage Clustering (arXiv:2210.13690) — bounded-cost online clustering for long local sessions.
• Who Said What? Recorder's On-Device Solution for Labeling Speakers (Google Research Blog).