Skip to content

feat: separate speakers in a meeting's microphone recording (D-007) - #2

Merged
nkeating-cmd merged 3 commits into
mainfrom
feat/diarise
Oct 16, 2026
Merged

nkeating-cmd merged 3 commits into
mainfrom
feat/diarise

Conversation

@nkeating-cmd

Copy link
Copy Markdown
Owner

What

Separate speakers for meetings held in a room, where everyone reaches the microphone. A new action on a meeting's Audio tab works out who said what from the microphone recording and labels each microphone line Speaker 1, Speaker 2 and so on. The labels reach the transcript view, copy, export, search, Ask and the notes model.

  • Engine (src-tauri/src/diarise.rs), D-007:
    • Runs on Kōrero's own ort 2.0.0-rc.12, so there is one ONNX runtime, shared with transcribe-rs.
    • CPU only, after the meeting.
    • pyannote segmentation-3.0 is read for speaker changes, not just speech, and re-anchored every 10 s window.
    • WeSpeaker ResNet34-LM embeddings, then whole-meeting clustering.
  • Models (D-008): downloaded on first use (about 32 MB) and kept only if their size and sha256 match the pinned values. Not bundled in the installer.
  • Data:
    • TranscriptSeg.speaker is optional, with a serde default, so every saved meeting loads unchanged.
    • Meetings gain speakersFound, speakerNames (for naming in M3) and mergedFrom.
  • Refused, with the reason shown:
    • imports;
    • meetings recorded before 1.26, whose lines have no timings;
    • combined meetings, whose lines come from two recordings.

Stacked on #1 (the brief, decision log and lessons log). Merge that first; this diff then shrinks to the feature.

Why

Bet B4 in docs/PRODUCT.md (in Now; product-check.py gate B4: PASS). Today a meeting's only "speakers" are the two capture channels, so everyone in a room is "You".

Evidence (main laptop, 16 Oct)

  • cargo test --lib: 264 passed, 0 failed (13 new in diarise).
  • bun test unit-tests: 92 passed (10 new in unit-tests/speakers.test.ts). These include checks that meetings without speakers give exactly the text they gave before.
  • tsc: clean. eslint: 0 errors, and no warnings in the touched files.
  • clippy: no findings in the new code (64 pre-existing elsewhere; the CI step is advisory).
  • diar_ami_der (ignored; run locally, 177 MB of AMI audio):
    • Mean DER 21.5% with a 0.25 s collar on four AMI far-field test meetings, against a bar of 25%. For reference, pyannote's own pipeline scores 22.4% strict on all 16 test meetings; this scores 26.3% strict on 4 of them.
    • The check's own scorer agrees with pyannote.metrics to within 0.2 points.
    • 92 minutes of audio separated in about 65 s.
  • cargo tree -i ort-sys: one ort.
  • gitleaks on all 5 commits: no leaks. No personal paths or hostnames in the added lines.
  • Independent review (read-only agent, briefed with every accepted cost):
    • No Critical or High findings, and no path that loses saved meetings.
    • Fixed in this PR: M1–M4, L1–L4, L6, L8, L10, L12 (see the second commit).

Not done yet

  • Not clicked through in the running app. The UI is type-checked and unit-tested, but no installed build has run it. Next: a seeded hand-over build (dev-session section 3).
  • Review follow-ups:
    • L5: overlapping same-speaker turns.
    • L7: the models are hashed on the async runtime on every run.
    • L9: the whole recording is held in memory, about 230 MB per hour.
    • L11: a line already open for editing when separation starts loses its edit.
    • L13: regenerate bindings.ts with tauri-specta. It was already stale; this PR adds one command by hand, matching the Rust exactly.
    • L14: CPU load if a recording starts during separation.
    • L15: verify the segmentation-3.0 (MIT) and kaldi-native-fbank (Apache-2.0) licences.
  • Still in B4: naming, merge and split (M3); splitting lines where the speaker changes mid-sentence; mirroring the two model files to a Kōrero release (D-008).
  • No version bump: 1.44 ships first.

Decision records: D-007, D-008 in docs/DECISIONS.md. Lessons: L-012 closed (check-added), L-013 added.

🤖 Generated with Claude Code

https://claude.ai/code/session_01M9niYsTjphhTSYCLhCwg5n

@nkeating-cmd
nkeating-cmd changed the base branch from docs/product-brief to main October 16, 2026 22:35
nkeating-cmd and others added 3 commits October 17, 2026 11:36
For meetings held in a room, where everyone reaches the microphone: a new
"Separate speakers" action on the Audio tab works out who said what and
labels each microphone line Speaker 1, Speaker 2 and so on. The labels reach
copy, export, search, Ask and the notes model through combine().

Engine (src-tauri/src/diarise.rs), on Kōrero's own ort 2.0.0-rc.12 (one ONNX
runtime, shared with transcribe-rs), CPU only, after the meeting:
- pyannote segmentation-3.0 read for speaker changes (powerset), re-anchored
  every 10 s window; WeSpeaker ResNet34-LM embeddings; whole-meeting
  average-linkage clustering on segments >= 1.5 s, clusters under 20 s of
  speech folded in, the rest attached by voice.
- Models download on first use (about 32 MB) and are checked by size and
  sha256 before use; a mismatch is discarded, never loaded.
- Microphone lines take the speaker they overlap most. A line where the
  speaker changes mid-sentence takes whoever said most of it (word-timestamp
  splitting is a later step).

Data: TranscriptSeg gains speaker (serde default, so every saved meeting
loads unchanged); meetings gain speakersFound and speakerNames (naming UI is
M3). One voice found: lines stay unlabelled. Editing a line is refused while
that meeting's speakers are being separated, so the result cannot overwrite it.

Evidence (legion, 16 Oct):
- cargo test --lib: 262 passed (11 new), 5 ignored. bun test unit-tests: 82
  passed. tsc: clean. eslint: 0 errors (3 pre-existing warnings elsewhere).
  clippy: no findings in the new code (64 pre-existing elsewhere; CI step
  is advisory).
- diar_ami_der (ignored; local, 177 MB of AMI audio): mean DER 21.9% with a
  0.25 s collar on four AMI far-field test meetings, under the 25% bar; the
  same output scores 22.0% in pyannote.metrics, which validates the check's
  own scorer. 92 minutes of audio separated in 65 s.
- D-008 (Accepted): the two speaker-separation models download on first
  use, pinned by size and sha256, instead of being bundled.
- L-012 closes as check-added: diar_ami_der guards the shipped models.
- L-013 (open): scripted edits normalised mixed line endings in
  src/bindings.ts and wrote garbled macrons from an ANSI-read script; both
  caught before any push.
- Brief: B4's first slice recorded. product-check.py: 0 FAIL, 1 WARN (O3).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M9niYsTjphhTSYCLhCwg5n
An independent read-only review of the first commit found no Critical or
High issues and no path that loses saved meetings. Fixed here:

- M1, M2: meetings whose lines can't be matched by time to one recording
  are refused, in Rust (timing_problem) and in the UI (separationBlocker),
  with the reason shown: pre-1.26 meetings (no line timings: re-transcribe
  first) and combined meetings (marked mergedFrom from now on; older ones
  are caught because their times run backwards at the join).
- M3, L6: model downloads have a 20 s connect and 10 min total timeout, so a
  stalled connection can't hold the one job slot; a failed download never
  leaves a .partial file behind.
- M4: Activity, Today and the top bar name the job "Separating speakers" or
  "Downloading the speaker models" with a percentage, not "Re-transcribing,
  part 1048576 of 5983836".
- L1: re-transcribing clears the stale speakersFound count.
- L2: combine() groups by (source, speaker), so meetings without speakers
  give exactly the text they gave before (tested, including two channels
  renamed to the same label).
- L3: turns join across the 61 ms gap between segmentation windows (tested).
- L4: a non-finite embedding is treated as none.
- L8: one separation at a time, enforced in Rust.
- L10: finding no voice at all gets its own message.
- L12: progress shows "Separating" once the download ends; download events
  are throttled to one per percent.

Evidence (legion, 16 Oct): cargo test --lib 264 passed (2 new); bun test
unit-tests 92 passed (10 new, unit-tests/speakers.test.ts); tsc clean;
eslint: 0 warnings in the touched files. diar_ami_der: mean DER 21.5%
(was 21.9%) with a 0.25 s collar, bar 25%.

Not fixed here (follow-ups in the PR): L5 overlapping same-speaker turns,
L7 hashing 32 MB on the async runtime per run, L9 the whole recording held
in memory (about 230 MB per hour), L11 a line already open for editing when
separation starts, L13 regenerate bindings.ts with tauri-specta, L14 CPU
load if a recording starts during separation, L15 two licences to verify.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M9niYsTjphhTSYCLhCwg5n
@nkeating-cmd
nkeating-cmd merged commit f32ae92 into main Oct 16, 2026
2 checks passed
@nkeating-cmd
nkeating-cmd deleted the feat/diarise branch October 16, 2026 22:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant