From 8b9205ce3857847162470200e55133ad8f6861f1 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 19:20:43 +0000 Subject: [PATCH 1/2] Overhaul speaker labeling: user-paced naming, split voices, clean speaker list MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Naming flow (was: auto-open + 120s auto-save countdown): - Remove the countdown and auto-open entirely — naming is now user-initiated via the menu bar row, orange badge, or a new Speakers-tab banner button. - Closing the Name Speakers window keeps candidates pending instead of force-skipping them; Skip All is the only path that persists placeholders. - onNamingRequired appends candidate batches (a second meeting no longer clobbers a pending batch) and onPipelineIdle preserves the .userAction badge while candidates are pending. Unplayable speakers (dead play buttons): - Candidates with no extractable clip never reach the naming prompt and no profile is created for them (placeholders are excluded from matching, so a clipless placeholder could never be identified by voice or by ear). - skipNaming no longer persists placeholders whose clips failed to persist. - SpeakerStore.pruneUnidentifiablePlaceholders() removes legacy clipless placeholder profiles at launch. - Roster-auto-named profiles now get voice clips extracted directly into speaker_clips/ instead of shipping without audio. Speakers table: - "Time in Meetings" column was missing its sort key path — now sortable. Split voices (merged diarization clusters): - Pipeline computes a per-clip embedding for every extracted sample from the diarizer's chunk embeddings (VAD-mapped overlap, aggregated via SpeakerEmbeddingAggregator), carried on NamingCandidate.clipEmbeddings. - The naming window offers "Split voices" on multi-sample candidates: AppModel.splitCandidate replaces the candidate with one sub-candidate per clip, each with its clip-local embedding (never the polluted centroid). Matching/database hygiene: - Naming a candidate with an existing profile's name merges embedding, clips and stats into that profile instead of creating a duplicate (SpeakerMatcher.addEmbedding, extracted from updateDatabase). - Removed the N:N roster ordered auto-assignment — pairing sorted names to cluster order mislabeled speakers; multiple unknowns become suggestions. - Removed dead code: namingDismissTask, showNamingPrompt, NamingCandidateRow. Tests: SpeakerStore prune, SpeakerMatcher.addEmbedding cap/replace/ignore. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01NUDNuK27wfK8FgajWw5wt1 --- Sources/Heard/MTApp.swift | 11 +- Sources/HeardCore/AppModel.swift | 116 +++++++++++++++----- Sources/HeardCore/AudioClipExtractor.swift | 18 +++- Sources/HeardCore/CoreModels.swift | 7 ++ Sources/HeardCore/MenuBarView.swift | 17 +-- Sources/HeardCore/Services.swift | 118 ++++++++++++++++++--- Sources/HeardCore/SettingsTabs.swift | 9 +- Sources/HeardCore/SettingsView.swift | 1 + Sources/HeardCore/SpeakerAssignment.swift | 25 +++-- Sources/HeardCore/SpeakerNamingView.swift | 90 ++++------------ Sources/HeardCore/Stores.swift | 25 +++++ Tests/HeardTests/TestRunner.swift | 50 +++++++++ handoff.md | 25 +++-- 13 files changed, 357 insertions(+), 155 deletions(-) diff --git a/Sources/Heard/MTApp.swift b/Sources/Heard/MTApp.swift index a18c628..81a29d9 100644 --- a/Sources/Heard/MTApp.swift +++ b/Sources/Heard/MTApp.swift @@ -108,13 +108,10 @@ struct HeardApp: App { SpeakerNamingView(model: appModel) .heardAppearance(appModel.settingsStore.settings.appearance) .onAppear { WindowActivationCoordinator.begin("speaker-naming") } - .onDisappear { - WindowActivationCoordinator.end("speaker-naming") - // If user closes window without naming, skip naming - if !appModel.namingCandidates.isEmpty { - appModel.skipNaming() - } - } + // Closing the window means "later", not "skip" — candidates stay + // pending and the window can be reopened from the menu bar. Skipping + // is only ever the explicit "Skip All" button. + .onDisappear { WindowActivationCoordinator.end("speaker-naming") } } .defaultSize(width: 560, height: 520) .windowResizability(.contentSize) diff --git a/Sources/HeardCore/AppModel.swift b/Sources/HeardCore/AppModel.swift index 716ef04..88f8c9d 100644 --- a/Sources/HeardCore/AppModel.swift +++ b/Sources/HeardCore/AppModel.swift @@ -7,8 +7,6 @@ public final class AppModel: ObservableObject { @Published public var phase: AppPhase = .dormant @Published public var errorMessage: String? @Published public var namingCandidates: [NamingCandidate] = [] - /// Set to true when naming prompt should be shown. Observed by the naming window scene. - @Published public var showNamingPrompt = false @Published public var selectedSettingsTab: SettingsTab = .general @Published public var speakerFilter = "" @Published public var vocabularyDraft = "" @@ -45,7 +43,6 @@ public final class AppModel: ObservableObject { private var cancellables = Set() private var stageWatchdogTimer: Timer? - private var namingDismissTask: Task? // 90 minutes is generous for any realistic workload (4-hour meeting transcription // on slow hardware). A genuine FluidAudio hang shows up well within this window. private static let maxStageSeconds: TimeInterval = 90 * 60 @@ -95,6 +92,11 @@ public final class AppModel: ObservableObject { // Archive speaker profiles inactive beyond the configured retention window speakerStore.archiveInactiveSpeakers(retentionDays: settingsStore.settings.speakerRetentionDays) + // Drop placeholder profiles whose voice clips no longer exist on disk — with + // neither a name nor audio they can never be identified, and they render as + // dead play buttons in the Speakers list. + speakerStore.pruneUnidentifiablePlaceholders() + // Clean stale recordings (>48h), preserving files referenced by active jobs let activeJobPaths = Set( queueStore.jobs @@ -189,15 +191,19 @@ public final class AppModel: ObservableObject { settingsStore: settingsStore, modelCatalog: modelCatalog, onNamingRequired: { [weak self] candidates in - NSLog("Heard: AppModel.onNamingRequired received \(candidates.count) candidate(s) — opening naming window") - self?.namingCandidates = candidates + NSLog("Heard: AppModel.onNamingRequired received \(candidates.count) candidate(s)") + // Append, don't replace — naming is user-paced (no auto-open, no + // countdown), so a batch from an earlier meeting may still be pending + // when the next meeting's pipeline finishes. + self?.namingCandidates.append(contentsOf: candidates) self?.phase = .userAction - self?.showNamingPrompt = true }, onPipelineIdle: { [weak self] in guard let self else { return } if self.phase == .processing { - self.phase = .dormant + // Keep the "Name Speakers" badge alive when candidates are still + // pending from this (or an earlier) meeting — naming is user-paced. + self.phase = self.namingCandidates.isEmpty ? .dormant : .userAction } } ) @@ -723,25 +729,51 @@ public var filteredSpeakers: [SpeakerProfile] { queueStore.remove(job) } + /// Cap on persisted voice clips per profile. Meetings keep contributing clips as + /// names are re-confirmed or profiles merged; beyond this, the oldest are deleted. + private static let maxStoredClipsPerSpeaker = 5 + public func saveSpeakerName(candidate: NamingCandidate, name: String) { let trimmed = name.trimmingCharacters(in: .whitespacesAndNewlines) guard !trimmed.isEmpty else { return } // Move the audio clips to their persistent home so they survive the 48-hour // recordings cleanup and can be replayed from the Speakers settings tab. let persistedClips = candidate.audioClipURLs.compactMap { AudioClipExtractor.persistClip($0) } - speakerStore.upsert( - SpeakerProfile( - id: UUID(), - name: trimmed, - embeddings: candidate.embedding.isEmpty ? [] : [candidate.embedding], - firstSeen: Date(), - lastSeen: Date(), - meetingCount: 1, - totalMeetingDuration: candidate.totalMeetingDuration, - totalWordCount: candidate.totalWordCount, - audioClipURLs: persistedClips + + if var existing = speakerStore.speakers.first(where: { + !SpeakerMatcher.isPlaceholderName($0.name) + && $0.name.caseInsensitiveCompare(trimmed) == .orderedSame + }) { + // The name already belongs to a known profile — this meeting's voice just + // didn't match it confidently (new headset, room echo, …). Fold the new + // embedding and clips into that profile instead of creating a duplicate + // row that would compete with it in future matching. + SpeakerMatcher.addEmbedding(candidate.embedding, to: &existing.embeddings) + let combinedClips = persistedClips + existing.audioClipURLs + existing.audioClipURLs = Array(combinedClips.prefix(Self.maxStoredClipsPerSpeaker)) + for dropped in combinedClips.dropFirst(Self.maxStoredClipsPerSpeaker) { + try? FileManager.default.removeItem(at: dropped) + } + existing.lastSeen = Date() + existing.meetingCount += 1 + existing.totalMeetingDuration += candidate.totalMeetingDuration + existing.totalWordCount += candidate.totalWordCount + speakerStore.upsert(existing) + } else { + speakerStore.upsert( + SpeakerProfile( + id: UUID(), + name: trimmed, + embeddings: candidate.embedding.isEmpty ? [] : [candidate.embedding], + firstSeen: Date(), + lastSeen: Date(), + meetingCount: 1, + totalMeetingDuration: candidate.totalMeetingDuration, + totalWordCount: candidate.totalWordCount, + audioClipURLs: persistedClips + ) ) - ) + } // Rewrite every transcript that references the placeholder. Speaker // numbers are globally unique, so this normally only touches the one // transcript in `candidate.transcriptPath`, but the helper also scans @@ -754,25 +786,49 @@ public var filteredSpeakers: [SpeakerProfile] { } namingCandidates.removeAll { $0.id == candidate.id } if namingCandidates.isEmpty { - showNamingPrompt = false phase = queueStore.processingJob == nil ? .dormant : .processing } } + /// Split a candidate whose samples reveal multiple distinct voices into one + /// sub-candidate per clip, each carrying its clip-local embedding, so the user can + /// name (or discard) each voice separately. The transcript keeps the original + /// placeholder label — once diarization has merged the voices into one cluster, + /// segment-level re-attribution is no longer possible. + public func splitCandidate(_ candidate: NamingCandidate) { + guard let index = namingCandidates.firstIndex(where: { $0.id == candidate.id }), + candidate.audioClipURLs.count >= 2 else { return } + let parts = candidate.audioClipURLs.indices.map { i in + NamingCandidate( + id: UUID(), + temporaryName: "\(candidate.temporaryName) · voice \(i + 1)", + suggestedName: nil, + audioClipURLs: [candidate.audioClipURLs[i]], + // Prefer the clip-local embedding; the cluster centroid is exactly the + // polluted mixture the user is splitting away from, so never fall back + // to it — an empty embedding (profile without voice matching) is the + // honest alternative. + embedding: i < candidate.clipEmbeddings.count ? candidate.clipEmbeddings[i] : [], + clipEmbeddings: i < candidate.clipEmbeddings.count ? [candidate.clipEmbeddings[i]] : [], + transcriptPath: nil, + totalMeetingDuration: candidate.totalMeetingDuration, + totalWordCount: 0 + ) + } + namingCandidates.replaceSubrange(index...index, with: parts) + } + /// Drop a candidate without creating a SpeakerProfile. Used when the user - /// listens to the clips and realizes diarization collapsed two voices into - /// one cluster — saving would poison the speaker database with a merged - /// embedding. The transcript keeps the placeholder ("Speaker N"). The - /// temporary clip files are deleted since they aren't going anywhere. + /// decides a candidate isn't worth keeping (crosstalk, music, a merged + /// cluster they don't want to split) — saving would clutter or poison the + /// speaker database. The transcript keeps the placeholder ("Speaker N"). + /// The temporary clip files are deleted since they aren't going anywhere. public func discardCandidate(_ candidate: NamingCandidate) { for url in candidate.audioClipURLs { try? FileManager.default.removeItem(at: url) } namingCandidates.removeAll { $0.id == candidate.id } if namingCandidates.isEmpty { - namingDismissTask?.cancel() - namingDismissTask = nil - showNamingPrompt = false phase = queueStore.processingJob == nil ? .dormant : .processing } } @@ -781,6 +837,11 @@ public var filteredSpeakers: [SpeakerProfile] { // Store remaining unnamed candidates with generic names (preserving embeddings + clips). for candidate in namingCandidates { let persistedClips = candidate.audioClipURLs.compactMap { AudioClipExtractor.persistClip($0) } + // A placeholder profile with no playable clip can never be identified — + // placeholders are excluded from voice matching, so without audio there is + // no path to a name, by voice or by ear. Persisting it would only clutter + // the Speakers list. + guard !persistedClips.isEmpty else { continue } speakerStore.upsert( SpeakerProfile( id: UUID(), @@ -796,7 +857,6 @@ public var filteredSpeakers: [SpeakerProfile] { ) } namingCandidates.removeAll() - showNamingPrompt = false // processingJob (not activeJob) — a stale .failed job in the queue must // not pin the phase at .processing after the prompt closes. phase = queueStore.processingJob == nil ? .dormant : .processing diff --git a/Sources/HeardCore/AudioClipExtractor.swift b/Sources/HeardCore/AudioClipExtractor.swift index 17c7c09..5782d06 100644 --- a/Sources/HeardCore/AudioClipExtractor.swift +++ b/Sources/HeardCore/AudioClipExtractor.swift @@ -210,6 +210,14 @@ public enum AudioClipExtractor { return clipped } + /// One extracted voice sample: the saved WAV plus the original-audio time region it + /// was cut from, so callers can correlate the clip back to diarizer chunk data. + public struct ExtractedClip { + public let url: URL + public let startTime: TimeInterval + public let endTime: TimeInterval + } + /// Extract clips for all unmatched speakers and return candidate info. /// Each speaker gets up to `clipsPerSpeaker` distinct samples saved to the recordings /// directory, ordered best-first. @@ -225,8 +233,8 @@ public enum AudioClipExtractor { outputDirectory: URL, clipsPerSpeaker: Int = 3, vadSpeechSegments: [(startTime: TimeInterval, endTime: TimeInterval)] = [] - ) -> [(temporaryName: String, clipURLs: [URL], embedding: [Float], duration: TimeInterval, words: Int)] { - var results: [(temporaryName: String, clipURLs: [URL], embedding: [Float], duration: TimeInterval, words: Int)] = [] + ) -> [(speakerID: String, temporaryName: String, clips: [ExtractedClip], embedding: [Float], duration: TimeInterval, words: Int)] { + var results: [(speakerID: String, temporaryName: String, clips: [ExtractedClip], embedding: [Float], duration: TimeInterval, words: Int)] = [] for speaker in unmatchedSpeakers { let regions = bestClipRegions( @@ -236,7 +244,7 @@ public enum AudioClipExtractor { maxCount: clipsPerSpeaker ) - var savedURLs: [URL] = [] + var saved: [ExtractedClip] = [] for region in regions { let clipFilename = "clip_\(UUID().uuidString.prefix(8)).wav" let clipURL = outputDirectory.appendingPathComponent(clipFilename) @@ -248,11 +256,11 @@ public enum AudioClipExtractor { outputURL: clipURL, vadSpeechSegments: vadSpeechSegments ) { - savedURLs.append(savedURL) + saved.append(ExtractedClip(url: savedURL, startTime: region.startTime, endTime: region.endTime)) } } - results.append((speaker.temporaryName, savedURLs, speaker.embedding, speaker.duration, speaker.words)) + results.append((speaker.speakerID, speaker.temporaryName, saved, speaker.embedding, speaker.duration, speaker.words)) } return results diff --git a/Sources/HeardCore/CoreModels.swift b/Sources/HeardCore/CoreModels.swift index d426297..1cb1b5d 100644 --- a/Sources/HeardCore/CoreModels.swift +++ b/Sources/HeardCore/CoreModels.swift @@ -228,6 +228,11 @@ public struct NamingCandidate: Identifiable, Equatable { /// crosstalk. public var audioClipURLs: [URL] public var embedding: [Float] + /// Per-clip embeddings parallel to `audioClipURLs` (an entry may be empty when the + /// diarizer didn't expose chunk embeddings for that region). Powers "Split voices": + /// when the clips of one cluster turn out to be different people, each split part is + /// saved with its own clip-local embedding instead of the polluted cluster centroid. + public var clipEmbeddings: [[Float]] /// Path to the transcript file that uses this temporary name; used to rewrite the file when the speaker is named. public var transcriptPath: URL? public var totalMeetingDuration: TimeInterval @@ -239,6 +244,7 @@ public struct NamingCandidate: Identifiable, Equatable { suggestedName: String? = nil, audioClipURLs: [URL] = [], embedding: [Float] = [], + clipEmbeddings: [[Float]] = [], transcriptPath: URL? = nil, totalMeetingDuration: TimeInterval = 0, totalWordCount: Int = 0 @@ -248,6 +254,7 @@ public struct NamingCandidate: Identifiable, Equatable { self.suggestedName = suggestedName self.audioClipURLs = audioClipURLs self.embedding = embedding + self.clipEmbeddings = clipEmbeddings self.transcriptPath = transcriptPath self.totalMeetingDuration = totalMeetingDuration self.totalWordCount = totalWordCount diff --git a/Sources/HeardCore/MenuBarView.swift b/Sources/HeardCore/MenuBarView.swift index c2fc568..5fe3d02 100644 --- a/Sources/HeardCore/MenuBarView.swift +++ b/Sources/HeardCore/MenuBarView.swift @@ -168,20 +168,9 @@ public struct MenuBarView: View { } .frame(width: 268) .background(HeardTheme.Paper.bg) - .onChange(of: model.showNamingPrompt) { _, show in - NSLog("Heard: MenuBarView observed showNamingPrompt=\(show)") - if show { - openWindow(id: "speaker-naming") - NSApp.activate(ignoringOtherApps: true) - } - } - .onChange(of: model.namingCandidates.isEmpty) { wasEmpty, isEmpty in - if wasEmpty && !isEmpty { - NSLog("Heard: MenuBarView observed namingCandidates became non-empty (\(model.namingCandidates.count))") - openWindow(id: "speaker-naming") - NSApp.activate(ignoringOtherApps: true) - } - } + // Naming is user-initiated: no auto-open. The orange badge, the + // "Name Speakers…" row above, and the Speakers-tab banner are the cues; + // candidates stay pending until the user acts. } // MARK: Status Header diff --git a/Sources/HeardCore/Services.swift b/Sources/HeardCore/Services.swift index 731443e..81b21f9 100644 --- a/Sources/HeardCore/Services.swift +++ b/Sources/HeardCore/Services.swift @@ -2514,22 +2514,37 @@ public final class PipelineProcessor: ObservableObject { vadSpeechSegments: vadSegments ) + // A candidate the user can't listen to can't be identified — and a + // placeholder profile is excluded from voice matching, so persisting one + // without audio would only clutter the Speakers list. Drop clipless + // speakers here; their transcript keeps the "Speaker N" label. + let audible = clips.filter { !$0.clips.isEmpty } + for silent in clips where silent.clips.isEmpty { + NSLog("Heard: Skipping naming candidate '\(silent.temporaryName)' — no playable clip could be extracted") + } + // Build candidates with audio clips, embeddings, and suggested roster names let rosterSuggestions = transcript.unmatchedRosterNames - let candidates = clips.enumerated().map { (index, clip) in + let candidates = audible.enumerated().map { (index, clip) in NamingCandidate( id: UUID(), temporaryName: clip.temporaryName, suggestedName: index < rosterSuggestions.count ? rosterSuggestions[index] : nil, - audioClipURLs: clip.clipURLs, + audioClipURLs: clip.clips.map(\.url), embedding: clip.embedding, + clipEmbeddings: perClipEmbeddings( + speakerID: clip.speakerID, + regions: clip.clips.map { (startTime: $0.startTime, endTime: $0.endTime) } + ), transcriptPath: outputURL, totalMeetingDuration: clip.duration, totalWordCount: clip.words ) } - NSLog("Heard: Triggering naming prompt for \(candidates.count) candidate(s)") - onNamingRequired(candidates) + if !candidates.isEmpty { + NSLog("Heard: Triggering naming prompt for \(candidates.count) candidate(s)") + onNamingRequired(candidates) + } } } } @@ -2863,6 +2878,76 @@ public final class PipelineProcessor: ObservableObject { return fallback } + /// One embedding per extracted voice clip, aggregated from the diarizer chunk + /// embeddings that overlap the clip's original-time region. Chunk times are in the + /// preprocessed (VAD-trimmed) timebase, so they're mapped through the app track's + /// `VadSegmentMap` before overlap is computed — the same mapping used for the + /// diarization segments themselves. Returns an empty vector for a clip when no + /// chunk overlaps it (or when chunk embeddings are unavailable entirely), so the + /// result is always parallel to `regions`. + private func perClipEmbeddings( + speakerID: String, + regions: [(startTime: TimeInterval, endTime: TimeInterval)] + ) -> [[Float]] { + guard let chunks = appDiarization?.chunkEmbeddings, !chunks.isEmpty else { + return regions.map { _ in [] } + } + let speakerChunks = chunks.filter { "R_\($0.speakerId)" == speakerID && !$0.embedding256.isEmpty } + guard !speakerChunks.isEmpty else { return regions.map { _ in [] } } + + let vadMap = appTrack?.vadMap + return regions.map { region in + var overlapping: [SpeakerEmbeddingAggregator.Chunk] = [] + for chunk in speakerChunks { + let start = vadMap?.toOriginalTime(TimeInterval(chunk.startTimeSeconds)) ?? TimeInterval(chunk.startTimeSeconds) + let end = vadMap?.toOriginalTime(TimeInterval(chunk.endTimeSeconds)) ?? TimeInterval(chunk.endTimeSeconds) + let overlap = min(end, region.endTime) - max(start, region.startTime) + if overlap > 0 { + overlapping.append(.init(vector: chunk.embedding256, weight: Float(overlap))) + } + } + return SpeakerEmbeddingAggregator.centroid(of: overlapping) ?? [] + } + } + + /// Extract up to two voice samples for a speaker directly into the persistent + /// `speaker_clips/` directory. Used for profiles created without going through the + /// naming prompt (roster auto-assignment), whose clips would otherwise never exist. + private func extractProfileClips( + speakerID: String, + diarizationSegments: [(speakerID: String, startTime: TimeInterval, endTime: TimeInterval)], + sourceAudioURL: URL + ) -> [URL] { + let speechSegments = appTrack?.vadMap.mappings.map { + (startTime: $0.originalStart, endTime: $0.originalEnd) + } + let regions = AudioClipExtractor.bestClipRegions( + speakerID: speakerID, + diarizationSegments: diarizationSegments, + speechSegments: speechSegments, + maxCount: 2 + ) + guard !regions.isEmpty else { return [] } + + let clipsDir = FileManager.default.heardSpeakerClipsDirectory + try? FileManager.default.createDirectory(at: clipsDir, withIntermediateDirectories: true) + + var saved: [URL] = [] + for region in regions { + let clipURL = clipsDir.appendingPathComponent("clip_\(UUID().uuidString.prefix(8)).wav") + if let url = AudioClipExtractor.extractClip( + from: sourceAudioURL, + startTime: region.startTime, + endTime: region.endTime, + outputURL: clipURL, + vadSpeechSegments: speechSegments ?? [] + ) { + saved.append(url) + } + } + return saved + } + // MARK: - Stage 4: Speaker Assignment private func runSpeakerAssignment(_ job: PipelineJob) -> TranscriptDocument { @@ -2954,15 +3039,12 @@ public final class PipelineProcessor: ObservableObject { let rosterName = unmatchedRosterNames.first! nameMap[speakerID] = rosterName NSLog("Heard: Auto-assigned roster name '\(rosterName)' to \(speakerID)") - } else if unmatchedSpeakers.count == unmatchedRosterNames.count && unmatchedSpeakers.count > 0 { - // Same number of unmatched speakers and roster names — assign in order - let sortedRoster = unmatchedRosterNames.sorted() - for (i, speaker) in unmatchedSpeakers.enumerated() where i < sortedRoster.count { - nameMap[speaker.detectedSpeakerID] = sortedRoster[i] - NSLog("Heard: Auto-assigned roster name '\(sortedRoster[i])' to \(speaker.detectedSpeakerID)") - } } else { - // Roster names available but count mismatch — pass as suggestions for naming prompt + // Two or more unknown voices: pairing sorted roster names to diarizer + // cluster order would be a coin flip that silently writes wrong + // name↔voice pairs into transcripts and poisons the speaker database + // with mislabeled embeddings. Pass the names as suggestions instead — + // the naming prompt lets the user confirm each one by ear. unmatchedRosterNamesForPrompt = unmatchedRosterNames.sorted() } } @@ -3001,13 +3083,23 @@ public final class PipelineProcessor: ObservableObject { && !match.embedding.isEmpty && !stillUnmatchedIDs.contains(match.detectedSpeakerID) { let resolvedName = nameMap[match.detectedSpeakerID] ?? match.assignedName + // These profiles bypass the naming prompt (which is where clips are + // normally attached), so extract a voice sample here — straight into + // the persistent speaker_clips directory — or the profile would sit + // in the Speakers list with a dead play button forever. + let clipURLs = extractProfileClips( + speakerID: match.detectedSpeakerID, + diarizationSegments: diarSegTuples, + sourceAudioURL: job.appAudioPath + ) let profile = SpeakerProfile( id: UUID(), name: resolvedName, embeddings: [match.embedding], firstSeen: Date(), lastSeen: Date(), - meetingCount: 1 + meetingCount: 1, + audioClipURLs: clipURLs ) speakerStore.upsert(profile) } diff --git a/Sources/HeardCore/SettingsTabs.swift b/Sources/HeardCore/SettingsTabs.swift index 720ec79..d352d4c 100644 --- a/Sources/HeardCore/SettingsTabs.swift +++ b/Sources/HeardCore/SettingsTabs.swift @@ -362,10 +362,15 @@ extension SettingsView { Image(systemName: "person.badge.plus") .foregroundStyle(HeardTheme.Paper.warn) .font(.system(size: 12)) - Text("New speakers detected — open the speaker naming window to identify them") + Text("\(model.namingCandidates.count) new speaker\(model.namingCandidates.count == 1 ? "" : "s") waiting to be named") .font(.system(size: 12)) .foregroundStyle(HeardTheme.Paper.warn) Spacer() + Button("Name Speakers…") { + openWindow(id: "speaker-naming") + NSApp.activate(ignoringOtherApps: true) + } + .controlSize(.small) } .padding(12) .background(HeardTheme.Paper.warnSoft, @@ -414,7 +419,7 @@ extension SettingsView { Text("\(speaker.meetingCount)").monospacedDigit() } .width(min: 60, ideal: 70, max: 90) - TableColumn("Time in Meetings") { speaker in + TableColumn("Time in Meetings", value: \.totalMeetingDuration) { speaker in let hours = Int(speaker.totalMeetingDuration) / 3600 let minutes = (Int(speaker.totalMeetingDuration) % 3600) / 60 if hours > 0 { diff --git a/Sources/HeardCore/SettingsView.swift b/Sources/HeardCore/SettingsView.swift index 31745ed..ffc8f5f 100644 --- a/Sources/HeardCore/SettingsView.swift +++ b/Sources/HeardCore/SettingsView.swift @@ -17,6 +17,7 @@ enum HotkeyTarget: Identifiable { @State var tableSortOrder: [KeyPathComparator] = [ KeyPathComparator(\.lastSeen, order: .reverse) ] + @Environment(\.openWindow) var openWindow public init(model: AppModel) { self.model = model diff --git a/Sources/HeardCore/SpeakerAssignment.swift b/Sources/HeardCore/SpeakerAssignment.swift index 21279d3..0d43396 100644 --- a/Sources/HeardCore/SpeakerAssignment.swift +++ b/Sources/HeardCore/SpeakerAssignment.swift @@ -61,7 +61,7 @@ public enum SpeakerMatcher { public static let maxEmbeddingsPerSpeaker = 5 /// True when `name` matches the auto-generated `Speaker N` placeholder pattern. - /// Placeholders come from skipped or auto-dismissed naming prompts; they are + /// Placeholders come from skipped naming prompts; they are /// excluded from matching so the user always gets another chance to name the /// speaker on a later meeting. public static func isPlaceholderName(_ name: String) -> Bool { @@ -208,20 +208,23 @@ public enum SpeakerMatcher { guard var profile = speakerStore.speakers.first(where: { $0.id == profileID }) else { continue } profile.lastSeen = Date() profile.meetingCount += 1 - - // Add embedding if we have room, keeping diverse set - if profile.embeddings.count < maxEmbeddingsPerSpeaker { - profile.embeddings.append(match.embedding) - } else { - // Replace the most similar existing embedding (least diverse) - if let replaceIndex = mostSimilarIndex(to: match.embedding, in: profile.embeddings) { - profile.embeddings[replaceIndex] = match.embedding - } - } + addEmbedding(match.embedding, to: &profile.embeddings) speakerStore.upsert(profile) } } + /// Insert a new embedding into a stored set, respecting the per-speaker cap and + /// keeping the set diverse: append while there's room, otherwise replace the most + /// similar (least diverse) existing embedding. Empty embeddings are ignored. + public static func addEmbedding(_ embedding: [Float], to embeddings: inout [[Float]]) { + guard !embedding.isEmpty else { return } + if embeddings.count < maxEmbeddingsPerSpeaker { + embeddings.append(embedding) + } else if let replaceIndex = mostSimilarIndex(to: embedding, in: embeddings) { + embeddings[replaceIndex] = embedding + } + } + /// Find the index of the stored embedding most similar to the new one. private static func mostSimilarIndex(to new: [Float], in stored: [[Float]]) -> Int? { guard !stored.isEmpty else { return nil } diff --git a/Sources/HeardCore/SpeakerNamingView.swift b/Sources/HeardCore/SpeakerNamingView.swift index 7dd1acd..07bb314 100644 --- a/Sources/HeardCore/SpeakerNamingView.swift +++ b/Sources/HeardCore/SpeakerNamingView.swift @@ -1,34 +1,6 @@ import AVFoundation import SwiftUI -struct NamingCandidateRow: View { - let candidate: NamingCandidate - let onSave: (String) -> Void - - @State private var draft = "" - - var body: some View { - HStack(spacing: HeardTheme.Spacing.sm) { - Text(candidate.temporaryName) - .font(.callout.weight(.medium)) - .foregroundStyle(HeardTheme.Paper.ink2) - .frame(width: 140, alignment: .leading) - TextField("Enter speaker name", text: $draft) - .textFieldStyle(.roundedBorder) - .onSubmit { save() } - Button("Save") { save() } - .buttonStyle(.borderedProminent) - .controlSize(.small) - .disabled(draft.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty) - } - } - - private func save() { - guard !draft.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty else { return } - onSave(draft) - } -} - // MARK: - Speaker Naming Prompt Window public struct SpeakerNamingView: View { @@ -37,8 +9,6 @@ public struct SpeakerNamingView: View { @State private var playingCandidateID: UUID? @State private var playingClipIndex: Int? @State private var audioPlayer: AVAudioPlayer? - @State private var countdownSeconds = 120 - @State private var countdownTask: Task? @Environment(\.dismissWindow) private var dismissWindow public init(model: AppModel) { self.model = model } @@ -59,16 +29,11 @@ public struct SpeakerNamingView: View { .font(.system(size: 17, weight: .semibold)) .foregroundStyle(HeardTheme.Paper.ink) - Text("Listen to each voice clip and enter their name. If a candidate's clips are clearly two different voices, choose Multiple speakers to drop it. Unnamed speakers will be saved with generic labels.") + Text("Listen to each voice clip and enter their name. If a candidate's samples are clearly different voices, use Split voices to name each one, or Discard to drop it. Take your time — nothing is saved until you choose. Closing this window keeps the speakers pending; reopen it anytime from the menu bar.") .font(.system(size: 12)) .foregroundStyle(HeardTheme.Paper.mute) .multilineTextAlignment(.center) .frame(maxWidth: 420) - - Text("Auto-saving in \(countdownSeconds)s") - .font(.system(size: 11)) - .foregroundStyle(HeardTheme.Paper.warn) - .monospacedDigit() } .padding(.top, HeardTheme.Spacing.lg) .padding(.bottom, HeardTheme.Spacing.md) @@ -111,15 +76,10 @@ public struct SpeakerNamingView: View { } .frame(width: 560) .background(HeardTheme.Paper.bg) - .onAppear { startCountdown() } - .onDisappear { - stopAudio() - countdownTask?.cancel() - } + .onDisappear { stopAudio() } .onChange(of: model.namingCandidates) { _, candidates in if candidates.isEmpty { stopAudio() - countdownTask?.cancel() dismissWindow(id: "speaker-naming") } } @@ -156,12 +116,20 @@ public struct SpeakerNamingView: View { .buttonStyle(.borderedProminent) .controlSize(.small) .disabled(draftText(for: candidate).isEmpty) - Button("Multiple speakers") { discardSingle(candidate) } + if candidate.audioClipURLs.count >= 2 { + Button("Split voices") { splitSingle(candidate) } + .buttonStyle(.plain) + .controlSize(.small) + .font(.system(size: 10)) + .foregroundStyle(HeardTheme.Paper.mute) + .help("The samples are different people? Split this candidate into one entry per sample so each voice can be named (or discarded) separately.") + } + Button("Discard") { discardSingle(candidate) } .buttonStyle(.plain) .controlSize(.small) .font(.system(size: 10)) .foregroundStyle(HeardTheme.Paper.mute) - .help("Drop this candidate without saving. Use when the clips reveal that diarization collapsed two voices into one — keeps the speaker database clean and leaves the transcript labeled Speaker N.") + .help("Drop this candidate without saving — keeps the speaker database clean and leaves the transcript labeled Speaker N.") } } .padding(HeardTheme.Spacing.md) @@ -172,7 +140,10 @@ public struct SpeakerNamingView: View { .stroke(HeardTheme.Paper.border, lineWidth: 0.5) ) .contextMenu { - Button("Multiple speakers — discard") { discardSingle(candidate) } + if candidate.audioClipURLs.count >= 2 { + Button("Split into separate voices") { splitSingle(candidate) } + } + Button("Discard candidate") { discardSingle(candidate) } } } @@ -262,10 +233,7 @@ public struct SpeakerNamingView: View { private func binding(for candidate: NamingCandidate) -> Binding { Binding( get: { drafts[candidate.id] ?? candidate.suggestedName ?? "" }, - set: { newValue in - drafts[candidate.id] = newValue - startCountdown() - } + set: { drafts[candidate.id] = $0 } ) } @@ -287,6 +255,12 @@ public struct SpeakerNamingView: View { drafts.removeValue(forKey: candidate.id) } + private func splitSingle(_ candidate: NamingCandidate) { + stopAudio() + model.splitCandidate(candidate) + drafts.removeValue(forKey: candidate.id) + } + private func saveAll() { for candidate in model.namingCandidates { let name = draftText(for: candidate) @@ -294,24 +268,6 @@ public struct SpeakerNamingView: View { } if !model.namingCandidates.isEmpty { model.skipNaming() } } - - // MARK: Countdown - - private func startCountdown() { - countdownSeconds = 120 - countdownTask?.cancel() - countdownTask = Task { - while !Task.isCancelled && countdownSeconds > 0 { - try? await Task.sleep(for: .seconds(1)) - guard !Task.isCancelled else { return } - countdownSeconds -= 1 - } - guard !Task.isCancelled else { return } - stopAudio() - saveAll() - dismissWindow(id: "speaker-naming") - } - } } // MARK: - Speaker Voice Cell (Speakers tab playback) diff --git a/Sources/HeardCore/Stores.swift b/Sources/HeardCore/Stores.swift index b67100f..11ac711 100644 --- a/Sources/HeardCore/Stores.swift +++ b/Sources/HeardCore/Stores.swift @@ -335,6 +335,31 @@ public final class SpeakerStore: ObservableObject { return stale.count } + /// Deletes placeholder-named profiles ("Speaker_XXXXXX") that have no playable + /// voice clip on disk. Placeholders are excluded from voice matching, so a profile + /// that offers neither a name nor audio can never be identified — it only adds + /// noise (and a dead play button) to the Speakers list. Returns the number removed. + @discardableResult + public func pruneUnidentifiablePlaceholders() -> Int { + let fm = FileManager.default + let doomed = speakers.filter { profile in + SpeakerMatcher.isPlaceholderName(profile.name) + && !profile.audioClipURLs.contains { fm.fileExists(atPath: $0.path) } + } + guard !doomed.isEmpty else { return 0 } + let doomedIDs = Set(doomed.map(\.id)) + for profile in doomed { + // Clip entries exist but the files are gone; removal is a no-op safety net. + for clipURL in profile.audioClipURLs { + try? fm.removeItem(at: clipURL) + } + } + speakers.removeAll { doomedIDs.contains($0.id) } + persist() + NSLog("Heard: pruned \(doomed.count) unidentifiable placeholder speaker(s)") + return doomed.count + } + public func merge(primaryID: UUID, secondaryID: UUID) { guard let primaryIndex = speakers.firstIndex(where: { $0.id == primaryID }), diff --git a/Tests/HeardTests/TestRunner.swift b/Tests/HeardTests/TestRunner.swift index 6e5a62f..c3e4b9a 100644 --- a/Tests/HeardTests/TestRunner.swift +++ b/Tests/HeardTests/TestRunner.swift @@ -786,6 +786,33 @@ func runTranscriptWriterTests() { try expect(store.speakers.isEmpty) } + test("SpeakerStore prunes placeholder profiles without playable clips") { + let tmpDir = FileManager.default.temporaryDirectory + .appendingPathComponent("HeardTests-\(UUID().uuidString)") + try FileManager.default.createDirectory(at: tmpDir, withIntermediateDirectories: true) + defer { try? FileManager.default.removeItem(at: tmpDir) } + + let existingClip = tmpDir.appendingPathComponent("clip.wav") + try Data([0x52, 0x49, 0x46, 0x46]).write(to: existingClip) + let missingClip = tmpDir.appendingPathComponent("gone.wav") + + func profile(_ name: String, clips: [URL]) -> SpeakerProfile { + SpeakerProfile(id: UUID(), name: name, embeddings: [], + firstSeen: Date(), lastSeen: Date(), meetingCount: 1, + audioClipURLs: clips) + } + + let store = SpeakerStore(url: tmpDir.appendingPathComponent("speakers.json")) + store.upsert(profile("Alice", clips: [])) // named, kept + store.upsert(profile("Speaker_AB12CD", clips: [existingClip])) // placeholder + audio, kept + store.upsert(profile("Speaker_EF34GH", clips: [missingClip])) // placeholder, clip gone + store.upsert(profile("Speaker_IJ56KL", clips: [])) // placeholder, never had audio + + let removed = store.pruneUnidentifiablePlaceholders() + try expectEqual(removed, 2) + try expectEqual(Set(store.speakers.map(\.name)), Set(["Alice", "Speaker_AB12CD"])) + } + test("PipelineQueueStore enqueue and retrieve") { let tmpDir = FileManager.default.temporaryDirectory .appendingPathComponent("HeardTests-\(UUID().uuidString)") @@ -1781,6 +1808,29 @@ func runSpeakerMatcherEdgeTests() { ) try expect(result[0].isNewSpeaker) } + + test("addEmbedding appends while under the per-speaker cap") { + var stored: [[Float]] = [ref] + SpeakerMatcher.addEmbedding(vector(atDistance: 0.20, from: ref), to: &stored) + try expectEqual(stored.count, 2) + } + + test("addEmbedding at cap replaces the most similar stored embedding") { + // Five stored embeddings at increasing distance from the reference axis. + var stored: [[Float]] = (0.. · voice N`), each carrying its clip-local embedding — never the polluted cluster centroid (empty if unavailable). The transcript keeps the original placeholder (segment-level re-attribution is impossible post-merge). "Discard" remains for hopeless candidates. +- **Naming a speaker with an existing profile's name now merges instead of duplicating**: `saveSpeakerName` folds the new embedding (via `SpeakerMatcher.addEmbedding`, extracted from `updateDatabase` and unit-tested) and clips into the case-insensitively matching non-placeholder profile, bumping stats; clips are capped at 5 per profile (oldest deleted from disk). +- **Roster N:N ordered auto-assignment removed.** Pairing sorted roster names to diarizer cluster order was a coin flip that wrote wrong name↔voice pairs into transcripts and poisoned profiles. Only the unambiguous 1:1 case still auto-assigns; multiple unknowns become suggestions in the naming prompt. +- Dead code removed: `AppModel.namingDismissTask` (never scheduled), `showNamingPrompt` (auto-open plumbing), unused `NamingCandidateRow` view. +- ⚠️ Written in a Linux container without a Swift toolchain — needs `swift build` + `swift run HeardTests` on macOS CI to verify. + **UI file split (tech debt):** the ~1.9 kLOC `Views.swift` was split by view area into `DesignSystem.swift`, `MenuBarView.swift`, `SpeakerNamingView.swift`, `SettingsView.swift`, `SettingsTabs.swift`, and `SettingsComponents.swift` (all in `Sources/HeardCore/`). Pure move — no logic or symbol changes; `swift build` is clean and all 181 tests pass. ## What's Working @@ -154,12 +164,12 @@ The dictation feature captures mic audio, transcribes in real-time, and injects - Menu bar icons are SF Symbols with symbol effects (`recordingtape`, `record.circle` + `.breathe`, `waveform` + `.variableColor`, `exclamationmark.circle.fill`, `person.crop.circle.badge.exclamationmark`) - **Menu bar reactivity fix**: `MenuBarView` holds direct `@ObservedObject` subscriptions to `queueStore`, `recordingManager`, `pipelineProcessor`, and `meetingDetector` — required because `MenuBarExtra(.window)` does not reliably re-render from forwarded child-store `objectWillChange` events. `MeetingDetector` is now an `ObservableObject` with `@Published isWatching`; `MenuBarIcon` subscribes directly so the paused/dimmed state reflects toggles immediately. `PipelineProcessor.runNextIfNeeded` also recovers orphaned non-terminal jobs (left in mid-stage when the processor is idle) by re-queuing them, charging a retry against the lifetime cap. - Settings window (880×600, opened via `@Environment(\.openWindow)`) with 6 tabs: **General** (launch at login, auto-watch, developer mode, custom vocabulary, output folder, permissions, meeting notes hotkey), **Transcription** (model download status, pipeline keep-alive, force-unload), **Dictation** (enable, push-to-talk, hotkey recorder, model keep-alive, custom formatting commands, live status), **Speakers** (your name, inline rename, merge, delete, search/sort), **Advanced** (diarization clustering threshold slider with "More speakers" ↔ "Fewer speakers" labels, live numeric readout, reset-to-default), **About** -- Standalone "Name Speakers" window scene (id `speaker-naming`, 560×520) with per-candidate audio playback, roster suggestions, and 120 s auto-dismiss +- Standalone "Name Speakers" window scene (id `speaker-naming`, 560×520) with per-candidate audio playback, roster suggestions, and per-candidate Split voices / Discard actions — user-paced, no countdown - Keyboard input works in Settings — `WindowActivationCoordinator` reference-counts `.accessory`/`.regular` transitions across the Settings and Name Speakers windows so closing one while the other is still open never steals keyboard focus - Output folder picker via `NSOpenPanel` - Custom vocabulary management lives in the General tab (add/remove terms, 3-char min, 50-term cap) — terms applied to both transcription and dictation via CTC boosting - Custom formatting commands live in the Dictation tab (map spoken phrases like "new paragraph" to written text like `\n\n`) — applied to both transcription and dictation via ITN rules -- Speaker table with inline rename, merge, delete (context menu), search, and sort (Name / Last Seen / Meeting Count); a leading **Voice** column has a play/stop button that replays the speaker's saved voice clip via a shared `SpeakerClipController` so only one clip plays at a time +- Speaker table with inline rename, merge, delete (context menu), search, and sort (Name / Meetings / Time in Meetings / Last Seen); a leading **Voice** column has a play/stop button that replays the speaker's saved voice clip via a shared `SpeakerClipController` so only one clip plays at a time - Model download status with progress bars and per-card download buttons, plus a "Download All Models" shortcut - Permission status with grant buttons and System Settings deep-links (Microphone + Screen Recording + Accessibility), surfaced inside the General tab - Launch at login via `SMAppService` @@ -173,16 +183,15 @@ The dictation feature captures mic audio, transcribes in real-time, and injects - Used for automatic speaker name assignment when diarization detects unmatched speakers ### Speaker Naming Prompt (Fully Working) -- Dedicated "Name Speakers" window opens automatically after a meeting when unmatched speakers are detected -- Each unmatched speaker shows a **playable audio clip** (~10s of their clearest speech from diarization) +- Dedicated "Name Speakers" window, **user-initiated** — opened from the menu bar dropdown ("Name Speakers…" row, orange badge) or the Speakers settings tab banner; it does not auto-open and there is no countdown. Candidates stay pending until the user saves, skips, splits, or discards them; closing the window keeps them pending. +- Each unmatched speaker shows **playable audio clips** (up to 3 × ~10s of their clearest speech from diarization). Speakers with no extractable clip never reach the prompt — a voice the user can't hear can't be identified. - Audio playback via `AVAudioPlayer` with play/stop toggle per speaker - Suggested names from Teams roster when available (shown as orange hint text) - Text fields pre-populated with roster suggestions for quick confirmation - **"Save & Close"** commits all entered names and dismisses the window via `dismissWindow(id: "speaker-naming")`; **"Skip All"** saves remaining unnamed speakers with `Speaker N` labels and dismisses -- **"Multiple speakers" discard**: a per-row button (and context-menu entry) drops the candidate without creating a `SpeakerProfile`, keeping the speaker database clean when diarization merged two voices into one cluster. Temporary audio clips are deleted immediately. The transcript retains the UUID placeholder label; the user can rename it manually in the Markdown file. -- 120-second auto-dismiss countdown — saves unnamed speakers with "Speaker N" labels -- Speaker profiles created with voice embeddings from diarization, enabling future recognition -- **No duplicate "Speaker N" profiles**: `SpeakerMatcher.updateDatabase` only refreshes already-matched profiles. Roster-auto-assigned new speakers get a profile with the resolved name in `runSpeakerAssignment`. Unresolved new speakers are persisted exactly once — by `saveSpeakerName` (real name) or `skipNaming`/auto-dismiss (`Speaker N`). +- **"Split voices"**: when a candidate's samples are different people (diarization merged two voices into one cluster), a per-row button splits it into one sub-candidate per clip, each carrying a clip-local embedding, so each voice can be named separately. **"Discard"** drops a candidate without creating a `SpeakerProfile` (clips deleted immediately; transcript keeps the placeholder label). +- Speaker profiles created with voice embeddings from diarization, enabling future recognition. Naming a candidate with an existing profile's name merges into that profile instead of duplicating it. +- **No duplicate "Speaker N" profiles**: `SpeakerMatcher.updateDatabase` only refreshes already-matched profiles. Roster-auto-assigned new speakers get a profile with the resolved name in `runSpeakerAssignment`. Unresolved new speakers are persisted exactly once — by `saveSpeakerName` (real name) or `skipNaming` (`Speaker N`, only when a clip persisted). - **Globally unique "Speaker N" numbering**: `SpeakerStore` previously used a monotonic counter, but now assigns unique `UUID`-based placeholders (e.g. `Speaker_1AB23C`). This avoids collisions entirely across installations and ensures renaming safely affects only the right transcripts. - **Transcript files are rewritten with real names**: `NamingCandidate` carries the meeting's `transcriptPath`. `saveSpeakerName` calls `TranscriptWriter.renameSpeakerInDirectory(_:from:to:)` to scan every `.md` in the configured output directory and rewrite both `**Speaker N:**` body tags and the `**Participants:**` header line. With UUID-based placeholder numbers this safely catches and updates old transcripts. - **Cumulative transcription stats**: Every time the pipeline successfully assigns speakers to speech segments, `SpeakerStore` accurately accumulates that speaker's `totalSpeechDuration` and `totalWordCount`, visible in the Settings tab. From fb20ffbb8c01051abc2e3f2d35dd93c109c0b6eb Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 19:48:57 +0000 Subject: [PATCH 2/2] Add speaking-time metric and implement remaining speaker-labeling improvements Speaking Time (new metric; Time in Meetings retained): - SpeakerProfile.totalSpeakingTime accumulates each speaker's summed transcript-segment durations, computed from pre-merge segments so merged blocks don't overcount silence between sentences. Flows through UnmatchedSpeaker -> NamingCandidate -> profile create/merge paths, and SpeakerStore.updateStats. New sortable "Speaking Time" table column; both duration columns share a durationText formatter (2h 05m / 14m / 38s). - Old speakers.json files decode with speaking time defaulting to 0. Pending naming candidates survive restart: - NamingCandidate is Codable; NamingCandidateStore persists the pending list (naming_candidates.json) on every change via didSet and restores it at bootstrap, pruning clips whose files are gone and dropping candidates left clipless. Pending clips are excluded from the 48h stale-recordings sweep. startWatching/stopWatching no longer clobber the restored .userAction badge. Voice match strictness is user-tunable: - AppSettings.speakerMatchThreshold (default 0.30, 0.15-0.45) feeds SpeakerMatcher.matchSpeakers(matchThreshold:); new Advanced -> Voice Matching slider card (Stricter <-> Looser, reset to default). Honest roster suggestions: - Candidates carry the full unmatched-roster list (suggestedNames) rendered as tappable chips that fill the name field, replacing the arbitrary index pairing; pre-fill only when exactly one name is unmatched. Split parts inherit the parent's suggestions. N-way merge: - Merge Selected accepts 2+ profiles; survivor picked by SpeakerMatcher.mergePrimary (named > placeholder, then meetings, then oldest). One consent dialog covers all named profiles being folded in. - SpeakerStore.merge caps merged embeddings (5) and clips (5, overflow files deleted) and sums speaking time. Cleanup: the 5-field unmatched-speaker tuple threaded through TranscriptDocument/AudioClipExtractor is now the UnmatchedSpeaker struct. Tests: updateStats speaking time, merge caps, NamingCandidateStore round-trip + clip pruning, custom match threshold, mergePrimary ordering. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01NUDNuK27wfK8FgajWw5wt1 --- Sources/HeardCore/AppModel.swift | 94 ++++++++++------ Sources/HeardCore/AudioClipExtractor.swift | 8 +- Sources/HeardCore/CoreModels.swift | 81 +++++++++++-- Sources/HeardCore/Services.swift | 57 +++++++--- Sources/HeardCore/SettingsTabs.swift | 69 ++++++++++-- Sources/HeardCore/SpeakerAssignment.swift | 28 ++++- Sources/HeardCore/SpeakerNamingView.swift | 48 ++++++-- Sources/HeardCore/Stores.swift | 72 +++++++++++- Tests/HeardTests/TestRunner.swift | 125 +++++++++++++++++++++ handoff.md | 20 +++- 10 files changed, 502 insertions(+), 100 deletions(-) diff --git a/Sources/HeardCore/AppModel.swift b/Sources/HeardCore/AppModel.swift index 88f8c9d..a4b9ee4 100644 --- a/Sources/HeardCore/AppModel.swift +++ b/Sources/HeardCore/AppModel.swift @@ -6,7 +6,12 @@ import Foundation public final class AppModel: ObservableObject { @Published public var phase: AppPhase = .dormant @Published public var errorMessage: String? - @Published public var namingCandidates: [NamingCandidate] = [] + @Published public var namingCandidates: [NamingCandidate] = [] { + // Naming is user-paced and may be deferred across sessions — mirror every + // change to disk so pending candidates survive an app restart. + didSet { namingCandidateStore.save(namingCandidates) } + } + let namingCandidateStore = NamingCandidateStore() @Published public var selectedSettingsTab: SettingsTab = .general @Published public var speakerFilter = "" @Published public var vocabularyDraft = "" @@ -97,12 +102,17 @@ public final class AppModel: ObservableObject { // dead play buttons in the Speakers list. speakerStore.pruneUnidentifiablePlaceholders() + // Restore naming candidates deferred from a previous session — loaded before + // the stale-recordings sweep so their clip files are preserved by it. + let pendingCandidates = model.namingCandidateStore.load() + // Clean stale recordings (>48h), preserving files referenced by active jobs + // and by pending naming candidates. let activeJobPaths = Set( queueStore.jobs .filter { $0.stage != .complete } .flatMap { [$0.appAudioPath, $0.micAudioPath] } - ) + ).union(pendingCandidates.flatMap(\.audioClipURLs)) TempFileCleanup.cleanStaleRecordings(activeJobPaths: activeJobPaths) // Destroy orphaned private aggregate devices left behind by previous crashes @@ -126,6 +136,14 @@ public final class AppModel: ObservableObject { model.startWatching() } + // Surface candidates deferred from a previous session: badge + menu row + // come back exactly as they were when the app quit. + if !pendingCandidates.isEmpty { + NSLog("Heard: restored \(pendingCandidates.count) pending naming candidate(s) from disk") + model.namingCandidates = pendingCandidates + model.phase = .userAction + } + // Retry failed jobs once on relaunch (handles transient failures from previous session) queueStore.prepareForResume() @@ -304,15 +322,18 @@ public var filteredSpeakers: [SpeakerProfile] { public func startWatching() { meetingDetector.startWatching() - phase = .dormant + // Don't clobber the "Name Speakers" badge — pending candidates outrank + // the plain watching state until the user resolves them. + if phase != .userAction { phase = .dormant } } public func stopWatching() { // stopWatching may synchronously fire onMeetingEnded (which sets phase = .processing). - // Only fall back to .dormant when no meeting was active. + // Only fall back to .dormant when no meeting was active, and preserve a + // pending "Name Speakers" badge. let phaseBefore = phase meetingDetector.stopWatching() - if phase == phaseBefore { + if phase == phaseBefore && phase != .userAction { phase = .dormant } } @@ -758,6 +779,7 @@ public var filteredSpeakers: [SpeakerProfile] { existing.meetingCount += 1 existing.totalMeetingDuration += candidate.totalMeetingDuration existing.totalWordCount += candidate.totalWordCount + existing.totalSpeakingTime += candidate.totalSpeakingTime speakerStore.upsert(existing) } else { speakerStore.upsert( @@ -770,6 +792,7 @@ public var filteredSpeakers: [SpeakerProfile] { meetingCount: 1, totalMeetingDuration: candidate.totalMeetingDuration, totalWordCount: candidate.totalWordCount, + totalSpeakingTime: candidate.totalSpeakingTime, audioClipURLs: persistedClips ) ) @@ -802,7 +825,8 @@ public var filteredSpeakers: [SpeakerProfile] { NamingCandidate( id: UUID(), temporaryName: "\(candidate.temporaryName) · voice \(i + 1)", - suggestedName: nil, + // Any of the cluster's roster suggestions could belong to any part. + suggestedNames: candidate.suggestedNames, audioClipURLs: [candidate.audioClipURLs[i]], // Prefer the clip-local embedding; the cluster centroid is exactly the // polluted mixture the user is splitting away from, so never fall back @@ -812,7 +836,10 @@ public var filteredSpeakers: [SpeakerProfile] { clipEmbeddings: i < candidate.clipEmbeddings.count ? [candidate.clipEmbeddings[i]] : [], transcriptPath: nil, totalMeetingDuration: candidate.totalMeetingDuration, - totalWordCount: 0 + // Words and speaking time measured the merged cluster — attributing + // them to any single part would double-count. + totalWordCount: 0, + totalSpeakingTime: 0 ) } namingCandidates.replaceSubrange(index...index, with: parts) @@ -852,6 +879,7 @@ public var filteredSpeakers: [SpeakerProfile] { meetingCount: 1, totalMeetingDuration: candidate.totalMeetingDuration, totalWordCount: candidate.totalWordCount, + totalSpeakingTime: candidate.totalSpeakingTime, audioClipURLs: persistedClips ) ) @@ -873,34 +901,32 @@ public var filteredSpeakers: [SpeakerProfile] { speakerStore.rename(id: id, to: trimmed) // Prompt the user before retroactively renaming - if askUserToUpdateTranscripts(oldName: oldName, newName: trimmed) { + if askUserToUpdateTranscripts(oldNames: [oldName], newName: trimmed) { rewriteSpeakerAcrossTranscripts(from: oldName, to: trimmed) } } + /// Merge every selected profile (2 or more) into one. The survivor is chosen by + /// `SpeakerMatcher.mergePrimary`: a human-given name beats a placeholder, then + /// most meetings, then oldest profile. public func mergeSelectedSpeakers() { - let ids = Array(mergeSelection) - guard ids.count == 2 else { return } - - // Prefer the human-given name: if ids[0] is a placeholder but ids[1] is not, - // swap so the real name survives as primary. - var primaryID = ids[0] - var secondaryID = ids[1] - if let p0 = speakerStore.speakers.first(where: { $0.id == ids[0] }), - let p1 = speakerStore.speakers.first(where: { $0.id == ids[1] }), - SpeakerMatcher.isPlaceholderName(p0.name) && !SpeakerMatcher.isPlaceholderName(p1.name) { - primaryID = ids[1] - secondaryID = ids[0] - } - - if let primary = speakerStore.speakers.first(where: { $0.id == primaryID }), - let secondary = speakerStore.speakers.first(where: { $0.id == secondaryID }) { - if askUserToUpdateTranscripts(oldName: secondary.name, newName: primary.name) { + let selected = speakerStore.speakers.filter { mergeSelection.contains($0.id) } + guard selected.count >= 2, let primary = SpeakerMatcher.mergePrimary(of: selected) else { return } + let secondaries = selected.filter { $0.id != primary.id } + + // One prompt covers every explicitly-named profile being folded in; + // placeholder names are rewritten without asking (nothing user-authored + // is being replaced). + let namedSecondaries = secondaries.map(\.name).filter { !SpeakerMatcher.isPlaceholderName($0) } + let rewriteNamed = namedSecondaries.isEmpty + || askUserToUpdateTranscripts(oldNames: namedSecondaries, newName: primary.name) + + for secondary in secondaries { + if SpeakerMatcher.isPlaceholderName(secondary.name) || rewriteNamed { rewriteSpeakerAcrossTranscripts(from: secondary.name, to: primary.name) } + speakerStore.merge(primaryID: primary.id, secondaryID: secondary.id) } - - speakerStore.merge(primaryID: primaryID, secondaryID: secondaryID) mergeSelection.removeAll() } @@ -921,20 +947,22 @@ public var filteredSpeakers: [SpeakerProfile] { } } - private func askUserToUpdateTranscripts(oldName: String, newName: String) -> Bool { - // If it's a generated placeholder, there's no need to ask, just do it. - // The user only needs to be asked if they are renaming an already explicitly named speaker. - if SpeakerMatcher.isPlaceholderName(oldName) { + private func askUserToUpdateTranscripts(oldNames: [String], newName: String) -> Bool { + // Generated placeholders don't need consent — nothing user-authored is + // being replaced. Only ask when an explicitly named speaker is affected. + let named = oldNames.filter { !SpeakerMatcher.isPlaceholderName($0) } + if named.isEmpty { return true } + let list = named.map { "'\($0)'" }.joined(separator: ", ") let alert = NSAlert() alert.messageText = "Update past transcripts?" - alert.informativeText = "Would you like to retroactively replace '\(oldName)' with '\(newName)' in all previously saved transcripts?" + alert.informativeText = "Would you like to retroactively replace \(list) with '\(newName)' in all previously saved transcripts?" alert.addButton(withTitle: "Update Transcripts") alert.addButton(withTitle: "Skip") alert.alertStyle = .informational - + NSApp.activate(ignoringOtherApps: true) let response = alert.runModal() return response == .alertFirstButtonReturn diff --git a/Sources/HeardCore/AudioClipExtractor.swift b/Sources/HeardCore/AudioClipExtractor.swift index 5782d06..7fa2252 100644 --- a/Sources/HeardCore/AudioClipExtractor.swift +++ b/Sources/HeardCore/AudioClipExtractor.swift @@ -226,15 +226,15 @@ public enum AudioClipExtractor { /// When provided, silence gaps within each extracted clip are skipped so every playback /// is continuous speech rather than a raw time slice that may include long pauses. public static func extractSpeakerClips( - unmatchedSpeakers: [(speakerID: String, temporaryName: String, embedding: [Float], duration: TimeInterval, words: Int)], + unmatchedSpeakers: [UnmatchedSpeaker], diarizationSegments: [(speakerID: String, startTime: TimeInterval, endTime: TimeInterval)], speechSegments: [(startTime: TimeInterval, endTime: TimeInterval)]? = nil, sourceAudioURL: URL, outputDirectory: URL, clipsPerSpeaker: Int = 3, vadSpeechSegments: [(startTime: TimeInterval, endTime: TimeInterval)] = [] - ) -> [(speakerID: String, temporaryName: String, clips: [ExtractedClip], embedding: [Float], duration: TimeInterval, words: Int)] { - var results: [(speakerID: String, temporaryName: String, clips: [ExtractedClip], embedding: [Float], duration: TimeInterval, words: Int)] = [] + ) -> [(speaker: UnmatchedSpeaker, clips: [ExtractedClip])] { + var results: [(speaker: UnmatchedSpeaker, clips: [ExtractedClip])] = [] for speaker in unmatchedSpeakers { let regions = bestClipRegions( @@ -260,7 +260,7 @@ public enum AudioClipExtractor { } } - results.append((speaker.speakerID, speaker.temporaryName, saved, speaker.embedding, speaker.duration, speaker.words)) + results.append((speaker, saved)) } return results diff --git a/Sources/HeardCore/CoreModels.swift b/Sources/HeardCore/CoreModels.swift index 1cb1b5d..0eb13dd 100644 --- a/Sources/HeardCore/CoreModels.swift +++ b/Sources/HeardCore/CoreModels.swift @@ -151,8 +151,13 @@ public struct SpeakerProfile: Codable, Identifiable, Equatable { public var firstSeen: Date public var lastSeen: Date public var meetingCount: Int + /// Total wall-clock duration of meetings this speaker attended. public var totalMeetingDuration: TimeInterval public var totalWordCount: Int + /// Cumulative time this speaker actually spent talking (sum of their + /// transcript segment durations), as opposed to `totalMeetingDuration` + /// which counts the whole meeting for every attendee. + public var totalSpeakingTime: TimeInterval /// Persisted voice samples for this speaker (used for replay in settings). /// Ordered best-first; multiple samples help the user disambiguate when one is silent. public var audioClipURLs: [URL] @@ -166,6 +171,7 @@ public struct SpeakerProfile: Codable, Identifiable, Equatable { meetingCount: Int, totalMeetingDuration: TimeInterval = 0, totalWordCount: Int = 0, + totalSpeakingTime: TimeInterval = 0, audioClipURLs: [URL] = [] ) { self.id = id @@ -176,12 +182,13 @@ public struct SpeakerProfile: Codable, Identifiable, Equatable { self.meetingCount = meetingCount self.totalMeetingDuration = totalMeetingDuration self.totalWordCount = totalWordCount + self.totalSpeakingTime = totalSpeakingTime self.audioClipURLs = audioClipURLs } private enum CodingKeys: String, CodingKey { case id, name, embeddings, firstSeen, lastSeen, meetingCount - case totalMeetingDuration, totalWordCount + case totalMeetingDuration, totalWordCount, totalSpeakingTime case audioClipURLs case audioClipURL // legacy single-URL field } @@ -196,6 +203,7 @@ public struct SpeakerProfile: Codable, Identifiable, Equatable { meetingCount = try c.decode(Int.self, forKey: .meetingCount) totalMeetingDuration = try c.decodeIfPresent(TimeInterval.self, forKey: .totalMeetingDuration) ?? 0 totalWordCount = try c.decodeIfPresent(Int.self, forKey: .totalWordCount) ?? 0 + totalSpeakingTime = try c.decodeIfPresent(TimeInterval.self, forKey: .totalSpeakingTime) ?? 0 if let urls = try c.decodeIfPresent([URL].self, forKey: .audioClipURLs) { audioClipURLs = urls } else if let legacy = try c.decodeIfPresent(URL.self, forKey: .audioClipURL) { @@ -215,14 +223,20 @@ public struct SpeakerProfile: Codable, Identifiable, Equatable { try c.encode(meetingCount, forKey: .meetingCount) try c.encode(totalMeetingDuration, forKey: .totalMeetingDuration) try c.encode(totalWordCount, forKey: .totalWordCount) + try c.encode(totalSpeakingTime, forKey: .totalSpeakingTime) try c.encode(audioClipURLs, forKey: .audioClipURLs) } } -public struct NamingCandidate: Identifiable, Equatable { +/// Codable so pending candidates survive an app restart (`NamingCandidateStore`). +public struct NamingCandidate: Identifiable, Equatable, Codable { public let id: UUID public var temporaryName: String - public var suggestedName: String? + /// Roster names not matched to any known speaker this meeting. All of them are + /// offered as tappable suggestions on every candidate — with several unknown + /// voices there is no reliable way to pre-pair a specific name to a specific + /// voice, so the user picks by ear. + public var suggestedNames: [String] /// Voice samples for this candidate, ordered best-first. The naming prompt lets the /// user play any of them so they can disambiguate when one sample is silent or has /// crosstalk. @@ -237,27 +251,37 @@ public struct NamingCandidate: Identifiable, Equatable { public var transcriptPath: URL? public var totalMeetingDuration: TimeInterval public var totalWordCount: Int + /// Time this voice actually spent talking (sum of its segment durations). + public var totalSpeakingTime: TimeInterval + + /// The one name worth pre-filling: only when the roster leaves exactly one + /// unmatched name is the pairing unambiguous enough to type it in for the user. + public var suggestedName: String? { + suggestedNames.count == 1 ? suggestedNames.first : nil + } public init( id: UUID, temporaryName: String, - suggestedName: String? = nil, + suggestedNames: [String] = [], audioClipURLs: [URL] = [], embedding: [Float] = [], clipEmbeddings: [[Float]] = [], transcriptPath: URL? = nil, totalMeetingDuration: TimeInterval = 0, - totalWordCount: Int = 0 + totalWordCount: Int = 0, + totalSpeakingTime: TimeInterval = 0 ) { self.id = id self.temporaryName = temporaryName - self.suggestedName = suggestedName + self.suggestedNames = suggestedNames self.audioClipURLs = audioClipURLs self.embedding = embedding self.clipEmbeddings = clipEmbeddings self.transcriptPath = transcriptPath self.totalMeetingDuration = totalMeetingDuration self.totalWordCount = totalWordCount + self.totalSpeakingTime = totalSpeakingTime } } @@ -339,6 +363,13 @@ public struct AppSettings: Codable, Equatable { /// tab, which is easier than recovering from a merged-embedding poisoning a /// profile. public var diarizationClusteringSimilarity: Double + /// Cosine *distance* below which a detected voice is considered the same person + /// as a stored profile (cross-meeting recognition). Lower = stricter: fewer + /// false matches against the wrong profile, but more "new speaker" naming + /// prompts for people already in the database; higher = looser. Distinct from + /// `diarizationClusteringSimilarity`, which separates voices *within* one + /// meeting. + public var speakerMatchThreshold: Double public var appearance: AppAppearance /// Controls whether preprocessing runs the app and mic tracks sequentially (low /// memory) or concurrently. `.auto` decides based on the machine's physical RAM; @@ -389,6 +420,7 @@ public struct AppSettings: Codable, Equatable { enableZoomDetection: true, enableWebexDetection: true, diarizationClusteringSimilarity: 0.65, + speakerMatchThreshold: 0.30, appearance: .system, memoryMode: .auto, speakerRetentionDays: 90, @@ -416,6 +448,7 @@ public struct AppSettings: Codable, Equatable { enableZoomDetection: Bool = true, enableWebexDetection: Bool = true, diarizationClusteringSimilarity: Double = 0.65, + speakerMatchThreshold: Double = 0.30, appearance: AppAppearance = .system, memoryMode: MemoryMode = .auto, speakerRetentionDays: Int = 90, @@ -441,6 +474,7 @@ public struct AppSettings: Codable, Equatable { self.enableZoomDetection = enableZoomDetection self.enableWebexDetection = enableWebexDetection self.diarizationClusteringSimilarity = diarizationClusteringSimilarity + self.speakerMatchThreshold = speakerMatchThreshold self.appearance = appearance self.memoryMode = memoryMode self.speakerRetentionDays = speakerRetentionDays @@ -565,14 +599,43 @@ public struct TranscriptSegment: Identifiable, Equatable { } } +/// A diarized voice with no confident match in the speaker database, carrying the +/// per-meeting stats that flow into the profile once the user names (or skips) it. +public struct UnmatchedSpeaker { + public let speakerID: String + public let temporaryName: String + public let embedding: [Float] + /// Wall-clock duration of the meeting this voice appeared in. + public var totalMeetingDuration: TimeInterval + public var totalWordCount: Int + /// Time this voice actually spent talking (sum of its segment durations). + public var totalSpeakingTime: TimeInterval + + public init( + speakerID: String, + temporaryName: String, + embedding: [Float], + totalMeetingDuration: TimeInterval = 0, + totalWordCount: Int = 0, + totalSpeakingTime: TimeInterval = 0 + ) { + self.speakerID = speakerID + self.temporaryName = temporaryName + self.embedding = embedding + self.totalMeetingDuration = totalMeetingDuration + self.totalWordCount = totalWordCount + self.totalSpeakingTime = totalSpeakingTime + } +} + public struct TranscriptDocument { public var title: String public var startTime: Date public var endTime: Date public var participants: [String] public var segments: [TranscriptSegment] - /// Unmatched speakers from diarization (speakerID, temporary name, embedding, totalMeetingDuration, totalWordCount). - public var unmatchedSpeakers: [(speakerID: String, temporaryName: String, embedding: [Float], duration: TimeInterval, words: Int)] + /// Unmatched speakers from diarization, pending the naming prompt. + public var unmatchedSpeakers: [UnmatchedSpeaker] /// Diarization segments with original-time timestamps for clip extraction. public var diarizationSegments: [(speakerID: String, startTime: TimeInterval, endTime: TimeInterval)] /// Roster names not matched to known speakers (potential suggested names). @@ -589,7 +652,7 @@ public struct TranscriptDocument { endTime: Date, participants: [String], segments: [TranscriptSegment], - unmatchedSpeakers: [(speakerID: String, temporaryName: String, embedding: [Float], duration: TimeInterval, words: Int)] = [], + unmatchedSpeakers: [UnmatchedSpeaker] = [], diarizationSegments: [(speakerID: String, startTime: TimeInterval, endTime: TimeInterval)] = [], unmatchedRosterNames: [String] = [], notes: [MeetingNote] = [], diff --git a/Sources/HeardCore/Services.swift b/Sources/HeardCore/Services.swift index 81b21f9..1c577fa 100644 --- a/Sources/HeardCore/Services.swift +++ b/Sources/HeardCore/Services.swift @@ -2520,25 +2520,29 @@ public final class PipelineProcessor: ObservableObject { // speakers here; their transcript keeps the "Speaker N" label. let audible = clips.filter { !$0.clips.isEmpty } for silent in clips where silent.clips.isEmpty { - NSLog("Heard: Skipping naming candidate '\(silent.temporaryName)' — no playable clip could be extracted") + NSLog("Heard: Skipping naming candidate '\(silent.speaker.temporaryName)' — no playable clip could be extracted") } - // Build candidates with audio clips, embeddings, and suggested roster names - let rosterSuggestions = transcript.unmatchedRosterNames - let candidates = audible.enumerated().map { (index, clip) in + // Build candidates with audio clips, embeddings, and roster suggestions. + // Every candidate gets the full unmatched-roster list — with several + // unknown voices there is no reliable way to pre-pair a specific name + // to a specific voice, so the prompt offers all of them as tappable + // chips and the user picks by ear. + let candidates = audible.map { item in NamingCandidate( id: UUID(), - temporaryName: clip.temporaryName, - suggestedName: index < rosterSuggestions.count ? rosterSuggestions[index] : nil, - audioClipURLs: clip.clips.map(\.url), - embedding: clip.embedding, + temporaryName: item.speaker.temporaryName, + suggestedNames: transcript.unmatchedRosterNames, + audioClipURLs: item.clips.map(\.url), + embedding: item.speaker.embedding, clipEmbeddings: perClipEmbeddings( - speakerID: clip.speakerID, - regions: clip.clips.map { (startTime: $0.startTime, endTime: $0.endTime) } + speakerID: item.speaker.speakerID, + regions: item.clips.map { (startTime: $0.startTime, endTime: $0.endTime) } ), transcriptPath: outputURL, - totalMeetingDuration: clip.duration, - totalWordCount: clip.words + totalMeetingDuration: item.speaker.totalMeetingDuration, + totalWordCount: item.speaker.totalWordCount, + totalSpeakingTime: item.speaker.totalSpeakingTime ) } if !candidates.isEmpty { @@ -2997,7 +3001,7 @@ public final class PipelineProcessor: ObservableObject { var allSegments: [TranscriptSegment] = appSegments + micSegments // Apply diarization speaker labels - var unmatchedSpeakerInfo: [(speakerID: String, temporaryName: String, embedding: [Float], duration: TimeInterval, words: Int)] = [] + var unmatchedSpeakerInfo: [UnmatchedSpeaker] = [] var diarSegTuples: [(speakerID: String, startTime: TimeInterval, endTime: TimeInterval)] = [] var unmatchedRosterNamesForPrompt: [String] = [] @@ -3016,7 +3020,8 @@ public final class PipelineProcessor: ObservableObject { let matches = SpeakerMatcher.matchSpeakers( embeddings: uniqueEmbeddings, database: speakerStore.speakers, - localUserName: me + localUserName: me, + matchThreshold: Float(settingsStore.settings.speakerMatchThreshold) ) var nameMap: [String: String] = [:] @@ -3052,7 +3057,11 @@ public final class PipelineProcessor: ObservableObject { // Collect unmatched speaker info for naming prompt let stillUnmatched = matches.filter { $0.isNewSpeaker && nameMap[$0.detectedSpeakerID]?.hasPrefix("Speaker_") ?? true } unmatchedSpeakerInfo = stillUnmatched.map { - (speakerID: $0.detectedSpeakerID, temporaryName: nameMap[$0.detectedSpeakerID] ?? $0.assignedName, embedding: $0.embedding, duration: 0.0, words: 0) + UnmatchedSpeaker( + speakerID: $0.detectedSpeakerID, + temporaryName: nameMap[$0.detectedSpeakerID] ?? $0.assignedName, + embedding: $0.embedding + ) } // Collect diarization segments with original-time timestamps for clip extraction @@ -3124,18 +3133,30 @@ public final class PipelineProcessor: ObservableObject { let words = segment.text.split(whereSeparator: { $0.isWhitespace }).count wordsPerSpeaker[segment.speaker, default: 0] += words } + // Speaking time from the pre-merge segments: mergeConsecutive extends a + // block's endTime across the silence between same-speaker sentences, so the + // merged segments would overcount. Sentence-level durations are the honest + // approximation of time actually spent talking. + var speakingPerSpeaker: [String: TimeInterval] = [:] + for segment in allSegments { + speakingPerSpeaker[segment.speaker, default: 0] += max(0, segment.endTime - segment.startTime) + } // Apply to existing known speakers (batched — one disk write) let statUpdates = speakerStore.speakers .filter { segmentSpeakers.contains($0.name) } - .map { (id: $0.id, addDuration: meetingDuration, addWords: wordsPerSpeaker[$0.name] ?? 0) } + .map { (id: $0.id, + addDuration: meetingDuration, + addWords: wordsPerSpeaker[$0.name] ?? 0, + addSpeaking: speakingPerSpeaker[$0.name] ?? 0) } speakerStore.updateStats(statUpdates) // Apply to unmatched speakers for i in unmatchedSpeakerInfo.indices { let name = unmatchedSpeakerInfo[i].temporaryName - unmatchedSpeakerInfo[i].duration = meetingDuration - unmatchedSpeakerInfo[i].words = wordsPerSpeaker[name] ?? 0 + unmatchedSpeakerInfo[i].totalMeetingDuration = meetingDuration + unmatchedSpeakerInfo[i].totalWordCount = wordsPerSpeaker[name] ?? 0 + unmatchedSpeakerInfo[i].totalSpeakingTime = speakingPerSpeaker[name] ?? 0 } let userName = settingsStore.settings.userName.trimmingCharacters(in: .whitespacesAndNewlines) diff --git a/Sources/HeardCore/SettingsTabs.swift b/Sources/HeardCore/SettingsTabs.swift index d352d4c..2c3c7d2 100644 --- a/Sources/HeardCore/SettingsTabs.swift +++ b/Sources/HeardCore/SettingsTabs.swift @@ -395,7 +395,7 @@ extension SettingsView { Spacer() Button("Merge Selected") { model.mergeSelectedSpeakers() } - .disabled(model.mergeSelection.count != 2) + .disabled(model.mergeSelection.count < 2) } } .padding(HeardTheme.Spacing.lg) @@ -420,13 +420,10 @@ extension SettingsView { } .width(min: 60, ideal: 70, max: 90) TableColumn("Time in Meetings", value: \.totalMeetingDuration) { speaker in - let hours = Int(speaker.totalMeetingDuration) / 3600 - let minutes = (Int(speaker.totalMeetingDuration) % 3600) / 60 - if hours > 0 { - Text("\(hours)h \(minutes)m").monospacedDigit() - } else { - Text("\(minutes)m").monospacedDigit() - } + Text(Self.durationText(speaker.totalMeetingDuration)).monospacedDigit() + } + TableColumn("Speaking Time", value: \.totalSpeakingTime) { speaker in + Text(Self.durationText(speaker.totalSpeakingTime)).monospacedDigit() } TableColumn("Last Seen", value: \.lastSeen) { speaker in Text(speaker.lastSeen.formatted(date: .abbreviated, time: .omitted)) @@ -469,6 +466,14 @@ extension SettingsView { .background(HeardTheme.Paper.bg) } + /// Compact duration for the speaker-stats columns: "2h 05m", "14m", "38s". + static func durationText(_ duration: TimeInterval) -> String { + let total = Int(duration) + if total >= 3600 { return String(format: "%dh %02dm", total / 3600, (total % 3600) / 60) } + if total >= 60 { return "\(total / 60)m" } + return "\(total)s" + } + // MARK: Advanced var advancedSection: some View { @@ -604,6 +609,54 @@ extension SettingsView { } } + sectionGroup("Voice Matching") { + SettingsCard { + CardRow(isLast: false) { + VStack(alignment: .leading, spacing: 8) { + HStack { + Text("Match strictness") + .font(.system(size: 12, weight: .medium)) + .foregroundStyle(HeardTheme.Paper.ink) + Spacer() + Text(String(format: "%.2f", model.settingsStore.settings.speakerMatchThreshold)) + .font(.system(size: 12, design: .monospaced)) + .foregroundStyle(HeardTheme.Paper.mute) + } + HStack(spacing: 8) { + Text("Stricter") + .font(.system(size: 10)) + .foregroundStyle(HeardTheme.Paper.mute) + Slider( + value: settingsBinding(\.speakerMatchThreshold), + in: 0.15...0.45, + step: 0.05 + ) + Text("Looser") + .font(.system(size: 10)) + .foregroundStyle(HeardTheme.Paper.mute) + } + } + } + CardRow(isLast: false) { + Text("How close a voice must be to a saved profile to be recognized as the same person across meetings. Stricter asks you to name people more often but almost never mislabels; looser recognizes more voices automatically at some risk of matching the wrong profile. Default: 0.30.") + .font(.system(size: 11)) + .foregroundStyle(HeardTheme.Paper.mute) + } + CardRow(isLast: true) { + HStack { + Spacer() + Button("Reset to Default") { + model.settingsStore.settings.speakerMatchThreshold = + AppSettings.default.speakerMatchThreshold + } + .buttonStyle(.plain) + .font(.system(size: 11, weight: .medium)) + .foregroundStyle(HeardTheme.Paper.accent) + } + } + } + } + sectionGroup("Memory") { SettingsCard { CardRow { diff --git a/Sources/HeardCore/SpeakerAssignment.swift b/Sources/HeardCore/SpeakerAssignment.swift index 0d43396..114b683 100644 --- a/Sources/HeardCore/SpeakerAssignment.swift +++ b/Sources/HeardCore/SpeakerAssignment.swift @@ -44,11 +44,12 @@ public struct TrackDiarizationResult { /// Uses cosine distance with configurable thresholds. public enum SpeakerMatcher { - /// Cosine distance threshold for matching (lower = more similar). + /// Default cosine distance threshold for matching (lower = more similar). /// 0.30 is on the strict side of typical WeSpeaker ranges — the previous 0.40 /// caused false-positive matches against unrelated existing profiles, so newly /// detected speakers were silently classified as already-known and the naming - /// prompt never fired. + /// prompt never fired. User-tunable via `AppSettings.speakerMatchThreshold` + /// (Advanced → Voice Matching). public static let matchThreshold: Float = 0.30 /// Minimum gap between best and second-best match to accept a match. @@ -87,7 +88,8 @@ public enum SpeakerMatcher { public static func matchSpeakers( embeddings: [SpeakerEmbedding], database: [SpeakerProfile], - localUserName: String + localUserName: String, + matchThreshold: Float = SpeakerMatcher.matchThreshold ) -> [MatchResult] { var results: [MatchResult] = [] var usedProfileIDs = Set() @@ -109,7 +111,8 @@ public enum SpeakerMatcher { let match = findBestMatch( embedding: detected.vector, database: database, - excludeIDs: usedProfileIDs + excludeIDs: usedProfileIDs, + matchThreshold: matchThreshold ) if let match { @@ -149,7 +152,8 @@ public enum SpeakerMatcher { private static func findBestMatch( embedding: [Float], database: [SpeakerProfile], - excludeIDs: Set + excludeIDs: Set, + matchThreshold: Float ) -> DatabaseMatch? { guard !embedding.isEmpty else { return nil } @@ -213,6 +217,20 @@ public enum SpeakerMatcher { } } + /// Pick the profile that should survive an N-way merge: a human-given name always + /// beats a placeholder; within the same tier, the profile seen in the most meetings + /// wins (its stats and embeddings are the best-established), with the oldest + /// `firstSeen` as the deterministic tiebreaker. + public static func mergePrimary(of profiles: [SpeakerProfile]) -> SpeakerProfile? { + profiles.min { a, b in + let aPlaceholder = isPlaceholderName(a.name) + let bPlaceholder = isPlaceholderName(b.name) + if aPlaceholder != bPlaceholder { return !aPlaceholder } + if a.meetingCount != b.meetingCount { return a.meetingCount > b.meetingCount } + return a.firstSeen < b.firstSeen + } + } + /// Insert a new embedding into a stored set, respecting the per-speaker cap and /// keeping the set diverse: append while there's room, otherwise replace the most /// similar (least diverse) existing embedding. Empty embeddings are ignored. diff --git a/Sources/HeardCore/SpeakerNamingView.swift b/Sources/HeardCore/SpeakerNamingView.swift index 07bb314..dbfb85c 100644 --- a/Sources/HeardCore/SpeakerNamingView.swift +++ b/Sources/HeardCore/SpeakerNamingView.swift @@ -90,18 +90,11 @@ public struct SpeakerNamingView: View { clipButtons(for: candidate) VStack(alignment: .leading, spacing: HeardTheme.Spacing.xs) { - HStack(spacing: 6) { - Text(candidate.temporaryName) - .font(.system(size: 12, weight: .medium)) - .foregroundStyle(HeardTheme.Paper.mute) - if let suggested = candidate.suggestedName { - Text("maybe \(suggested)?") - .font(.system(size: 11)) - .foregroundStyle(HeardTheme.Paper.warn) - .padding(.horizontal, 6) - .padding(.vertical, 2) - .background(HeardTheme.Paper.warnSoft, in: Capsule()) - } + Text(candidate.temporaryName) + .font(.system(size: 12, weight: .medium)) + .foregroundStyle(HeardTheme.Paper.mute) + if !candidate.suggestedNames.isEmpty { + suggestionChips(for: candidate) } TextField( candidate.suggestedName ?? "Enter speaker name", @@ -147,6 +140,37 @@ public struct SpeakerNamingView: View { } } + /// Tappable roster-name chips. Every unmatched roster name is offered on every + /// candidate — with several unknown voices there is no reliable way to pre-pair + /// a specific name to a specific voice, so the user picks by ear. Tapping a chip + /// fills the name field; Save still commits. + private func suggestionChips(for candidate: NamingCandidate) -> some View { + HStack(spacing: 4) { + Text("maybe:") + .font(.system(size: 10)) + .foregroundStyle(HeardTheme.Paper.mute) + ForEach(candidate.suggestedNames.prefix(3), id: \.self) { name in + Button { + drafts[candidate.id] = name + } label: { + Text(name) + .font(.system(size: 11)) + .foregroundStyle(HeardTheme.Paper.warn) + .padding(.horizontal, 6) + .padding(.vertical, 2) + .background(HeardTheme.Paper.warnSoft, in: Capsule()) + } + .buttonStyle(.plain) + .help("Fill the name field with \(name)") + } + if candidate.suggestedNames.count > 3 { + Text("+\(candidate.suggestedNames.count - 3) more") + .font(.system(size: 10)) + .foregroundStyle(HeardTheme.Paper.mute) + } + } + } + @ViewBuilder private func clipButtons(for candidate: NamingCandidate) -> some View { if candidate.audioClipURLs.isEmpty { diff --git a/Sources/HeardCore/Stores.swift b/Sources/HeardCore/Stores.swift index 11ac711..911e618 100644 --- a/Sources/HeardCore/Stores.swift +++ b/Sources/HeardCore/Stores.swift @@ -34,6 +34,10 @@ public enum AppPaths { public static var speakersFile: URL { FileManager.default.heardAppSupportDirectory.appendingPathComponent("speakers.json") } + + public static var namingCandidatesFile: URL { + FileManager.default.heardAppSupportDirectory.appendingPathComponent("naming_candidates.json") + } } public enum Formatting { @@ -173,6 +177,11 @@ public final class SettingsStore: ObservableObject { diarizationClusteringSimilarity = val.doubleValue } + var speakerMatchThreshold = base.speakerMatchThreshold + if let val = defaults.object(forKey: "speakerMatchThreshold") as? NSNumber { + speakerMatchThreshold = val.doubleValue + } + var speakerRetentionDays = base.speakerRetentionDays if let val = defaults.object(forKey: "speakerRetentionDays") as? NSNumber { speakerRetentionDays = val.intValue @@ -207,6 +216,7 @@ public final class SettingsStore: ObservableObject { enableZoomDetection: defaults.object(forKey: "enableZoomDetection") as? Bool ?? base.enableZoomDetection, enableWebexDetection: defaults.object(forKey: "enableWebexDetection") as? Bool ?? base.enableWebexDetection, diarizationClusteringSimilarity: diarizationClusteringSimilarity, + speakerMatchThreshold: speakerMatchThreshold, memoryMode: memoryMode, speakerRetentionDays: speakerRetentionDays, selectedInputDeviceUID: defaults.string(forKey: "selectedInputDeviceUID"), @@ -242,6 +252,7 @@ public final class SettingsStore: ObservableObject { defaults.set(settings.enableZoomDetection, forKey: "enableZoomDetection") defaults.set(settings.enableWebexDetection, forKey: "enableWebexDetection") defaults.set(settings.diarizationClusteringSimilarity, forKey: "diarizationClusteringSimilarity") + defaults.set(settings.speakerMatchThreshold, forKey: "speakerMatchThreshold") defaults.set(settings.speakerRetentionDays, forKey: "speakerRetentionDays") defaults.set(settings.memoryMode.rawValue, forKey: "memoryMode") defaults.set(settings.showAdvancedSettings, forKey: "showAdvancedSettings") @@ -287,18 +298,19 @@ public final class SpeakerStore: ObservableObject { persist() } - public func updateStats(id: UUID, addDuration: TimeInterval, addWords: Int) { - updateStats([(id: id, addDuration: addDuration, addWords: addWords)]) + public func updateStats(id: UUID, addDuration: TimeInterval, addWords: Int, addSpeaking: TimeInterval = 0) { + updateStats([(id: id, addDuration: addDuration, addWords: addWords, addSpeaking: addSpeaking)]) } /// Batched variant: one disk write for the whole meeting's stat updates /// instead of a full JSON rewrite per speaker. - public func updateStats(_ updates: [(id: UUID, addDuration: TimeInterval, addWords: Int)]) { + public func updateStats(_ updates: [(id: UUID, addDuration: TimeInterval, addWords: Int, addSpeaking: TimeInterval)]) { var changed = false for update in updates { guard let index = speakers.firstIndex(where: { $0.id == update.id }) else { continue } speakers[index].totalMeetingDuration += update.addDuration speakers[index].totalWordCount += update.addWords + speakers[index].totalSpeakingTime += update.addSpeaking changed = true } if changed { persist() } @@ -360,6 +372,13 @@ public final class SpeakerStore: ObservableObject { return doomed.count } + /// Cap on embeddings and voice clips a merged profile may accumulate. Matching + /// scans every stored embedding, so unbounded growth from repeated (or N-way) + /// merges would slow matching and raise false-match risk; overflow clips are + /// deleted from disk. + static let maxMergedEmbeddings = SpeakerMatcher.maxEmbeddingsPerSpeaker + static let maxMergedClips = 5 + public func merge(primaryID: UUID, secondaryID: UUID) { guard let primaryIndex = speakers.firstIndex(where: { $0.id == primaryID }), @@ -369,13 +388,19 @@ public final class SpeakerStore: ObservableObject { var primary = speakers[primaryIndex] let secondary = speakers[secondaryIndex] - primary.embeddings.append(contentsOf: secondary.embeddings) - primary.audioClipURLs.append(contentsOf: secondary.audioClipURLs) + // Keep the primary's entries first — they belong to the surviving identity. + primary.embeddings = Array((primary.embeddings + secondary.embeddings).prefix(Self.maxMergedEmbeddings)) + let combinedClips = primary.audioClipURLs + secondary.audioClipURLs + primary.audioClipURLs = Array(combinedClips.prefix(Self.maxMergedClips)) + for dropped in combinedClips.dropFirst(Self.maxMergedClips) { + try? FileManager.default.removeItem(at: dropped) + } primary.firstSeen = min(primary.firstSeen, secondary.firstSeen) primary.lastSeen = max(primary.lastSeen, secondary.lastSeen) primary.meetingCount += secondary.meetingCount primary.totalMeetingDuration += secondary.totalMeetingDuration primary.totalWordCount += secondary.totalWordCount + primary.totalSpeakingTime += secondary.totalSpeakingTime speakers[primaryIndex] = primary speakers.remove(at: secondaryIndex) persist() @@ -409,6 +434,43 @@ public final class SpeakerStore: ObservableObject { } } +/// Persists pending naming candidates so they survive an app restart — naming is +/// user-paced and may be deferred across sessions. Loading drops clips whose files +/// are gone (48-hour recordings cleanup, crashes) and whole candidates left with no +/// playable clip, mirroring the pipeline's "can't listen → can't identify" rule. +@MainActor +public final class NamingCandidateStore { + private let store = JSONStore() + private let url: URL + + public init(url: URL = AppPaths.namingCandidatesFile) { + self.url = url + } + + public func load() -> [NamingCandidate] { + let raw = store.load([NamingCandidate].self, from: url, defaultValue: []) + let fm = FileManager.default + return raw.compactMap { candidate in + var kept = candidate + // Keep clipEmbeddings parallel to the surviving clips. + let pairs = candidate.audioClipURLs.enumerated().filter { fm.fileExists(atPath: $0.element.path) } + kept.audioClipURLs = pairs.map { $0.element } + kept.clipEmbeddings = pairs.compactMap { index, _ in + index < candidate.clipEmbeddings.count ? candidate.clipEmbeddings[index] : nil + } + return kept.audioClipURLs.isEmpty ? nil : kept + } + } + + public func save(_ candidates: [NamingCandidate]) { + if candidates.isEmpty { + try? FileManager.default.removeItem(at: url) + } else { + try? store.save(candidates, to: url) + } + } +} + @MainActor public final class PipelineQueueStore: ObservableObject { @Published public private(set) var jobs: [PipelineJob] diff --git a/Tests/HeardTests/TestRunner.swift b/Tests/HeardTests/TestRunner.swift index c3e4b9a..2144b79 100644 --- a/Tests/HeardTests/TestRunner.swift +++ b/Tests/HeardTests/TestRunner.swift @@ -813,6 +813,87 @@ func runTranscriptWriterTests() { try expectEqual(Set(store.speakers.map(\.name)), Set(["Alice", "Speaker_AB12CD"])) } + test("SpeakerStore updateStats accumulates speaking time") { + let tmpDir = FileManager.default.temporaryDirectory + .appendingPathComponent("HeardTests-\(UUID().uuidString)") + try FileManager.default.createDirectory(at: tmpDir, withIntermediateDirectories: true) + defer { try? FileManager.default.removeItem(at: tmpDir) } + + let store = SpeakerStore(url: tmpDir.appendingPathComponent("speakers.json")) + let id = UUID() + store.upsert(SpeakerProfile(id: id, name: "Alice", embeddings: [], firstSeen: Date(), lastSeen: Date(), meetingCount: 1)) + store.updateStats([(id: id, addDuration: 3600, addWords: 500, addSpeaking: 240)]) + store.updateStats([(id: id, addDuration: 1800, addWords: 100, addSpeaking: 60)]) + let alice = try unwrap(store.speakers.first) + try expectClose(alice.totalMeetingDuration, 5400) + try expectEqual(alice.totalWordCount, 600) + try expectClose(alice.totalSpeakingTime, 300) + } + + test("SpeakerStore merge caps embeddings and sums speaking time") { + let tmpDir = FileManager.default.temporaryDirectory + .appendingPathComponent("HeardTests-\(UUID().uuidString)") + try FileManager.default.createDirectory(at: tmpDir, withIntermediateDirectories: true) + defer { try? FileManager.default.removeItem(at: tmpDir) } + + let store = SpeakerStore(url: tmpDir.appendingPathComponent("speakers.json")) + let id1 = UUID(), id2 = UUID() + let fourVectors: [[Float]] = [[1, 0], [0, 1], [0.5, 0.5], [0.2, 0.8]] + store.upsert(SpeakerProfile(id: id1, name: "Alice", embeddings: fourVectors, + firstSeen: Date(), lastSeen: Date(), meetingCount: 1, + totalSpeakingTime: 100)) + store.upsert(SpeakerProfile(id: id2, name: "Dup", embeddings: fourVectors, + firstSeen: Date(), lastSeen: Date(), meetingCount: 1, + totalSpeakingTime: 50)) + store.merge(primaryID: id1, secondaryID: id2) + let merged = try unwrap(store.speakers.first) + try expectEqual(merged.embeddings.count, SpeakerMatcher.maxEmbeddingsPerSpeaker, + "8 combined embeddings capped at the per-speaker limit") + try expectClose(merged.totalSpeakingTime, 150) + } + + test("NamingCandidateStore round-trips candidates and drops missing clips") { + let tmpDir = FileManager.default.temporaryDirectory + .appendingPathComponent("HeardTests-\(UUID().uuidString)") + try FileManager.default.createDirectory(at: tmpDir, withIntermediateDirectories: true) + defer { try? FileManager.default.removeItem(at: tmpDir) } + + let existingClip = tmpDir.appendingPathComponent("clip.wav") + try Data([0x52, 0x49, 0x46, 0x46]).write(to: existingClip) + let missingClip = tmpDir.appendingPathComponent("gone.wav") + + let url = tmpDir.appendingPathComponent("naming_candidates.json") + let store = NamingCandidateStore(url: url) + store.save([ + NamingCandidate( + id: UUID(), temporaryName: "Speaker_AB12CD", + suggestedNames: ["Alice", "Bob"], + audioClipURLs: [missingClip, existingClip], + embedding: [1, 0], + clipEmbeddings: [[0.9, 0.1], [0.1, 0.9]], + totalSpeakingTime: 42 + ), + NamingCandidate( + id: UUID(), temporaryName: "Speaker_EF34GH", + audioClipURLs: [missingClip] + ), + ]) + + let loaded = store.load() + try expectEqual(loaded.count, 1, "Candidate with no surviving clip is dropped") + let survivor = try unwrap(loaded.first) + try expectEqual(survivor.temporaryName, "Speaker_AB12CD") + try expectEqual(survivor.suggestedNames, ["Alice", "Bob"]) + try expectEqual(survivor.audioClipURLs, [existingClip], "Missing clip pruned") + try expectEqual(survivor.clipEmbeddings, [[0.1, 0.9]], "Clip embeddings stay parallel to surviving clips") + try expectClose(survivor.totalSpeakingTime, 42) + + // Saving an empty list clears the file entirely. + store.save([]) + try expect(!FileManager.default.fileExists(atPath: url.path)) + try expect(store.load().isEmpty) + } + test("PipelineQueueStore enqueue and retrieve") { let tmpDir = FileManager.default.temporaryDirectory .appendingPathComponent("HeardTests-\(UUID().uuidString)") @@ -1831,6 +1912,50 @@ func runSpeakerMatcherEdgeTests() { SpeakerMatcher.addEmbedding([], to: &stored) try expectEqual(stored.count, 1) } + + test("Custom matchThreshold overrides the default") { + // Distance 0.35 fails the default 0.30 threshold but passes a looser 0.40. + let bob = profile("Bob", [ref]) + let probe = vector(atDistance: 0.35, from: ref) + let strict = SpeakerMatcher.matchSpeakers( + embeddings: [SpeakerEmbedding(speakerID: "R_0", vector: probe)], + database: [bob], + localUserName: "Me" + ) + try expect(strict[0].isNewSpeaker, "0.35 must not match at the default 0.30") + let loose = SpeakerMatcher.matchSpeakers( + embeddings: [SpeakerEmbedding(speakerID: "R_0", vector: probe)], + database: [bob], + localUserName: "Me", + matchThreshold: 0.40 + ) + try expectEqual(loose[0].assignedName, "Bob") + } + + test("mergePrimary prefers named over placeholder, then meetings, then age") { + func profileAt(_ name: String, meetings: Int, daysAgo: Double) -> SpeakerProfile { + SpeakerProfile( + id: UUID(), name: name, embeddings: [], + firstSeen: Date().addingTimeInterval(-daysAgo * 86400), + lastSeen: Date(), meetingCount: meetings + ) + } + let placeholder = profileAt("Speaker_AB12CD", meetings: 9, daysAgo: 300) + let newBob = profileAt("Bob", meetings: 1, daysAgo: 1) + let seasonedAlice = profileAt("Alice", meetings: 5, daysAgo: 30) + let olderAliceTie = profileAt("Alice2", meetings: 5, daysAgo: 90) + + try expectEqual( + SpeakerMatcher.mergePrimary(of: [placeholder, newBob])?.name, "Bob", + "Human name beats placeholder regardless of meeting count") + try expectEqual( + SpeakerMatcher.mergePrimary(of: [newBob, seasonedAlice])?.name, "Alice", + "More meetings wins between named profiles") + try expectEqual( + SpeakerMatcher.mergePrimary(of: [seasonedAlice, olderAliceTie])?.name, "Alice2", + "Meeting-count tie broken by oldest firstSeen") + try expect(SpeakerMatcher.mergePrimary(of: []) == nil) + } } // MARK: - RosterReader Tests diff --git a/handoff.md b/handoff.md index 9ce3620..73d839f 100644 --- a/handoff.md +++ b/handoff.md @@ -77,7 +77,16 @@ All items from the post-v0.2.2 robustness review are now resolved (see `ROADMAP. - **Naming a speaker with an existing profile's name now merges instead of duplicating**: `saveSpeakerName` folds the new embedding (via `SpeakerMatcher.addEmbedding`, extracted from `updateDatabase` and unit-tested) and clips into the case-insensitively matching non-placeholder profile, bumping stats; clips are capped at 5 per profile (oldest deleted from disk). - **Roster N:N ordered auto-assignment removed.** Pairing sorted roster names to diarizer cluster order was a coin flip that wrote wrong name↔voice pairs into transcripts and poisoned profiles. Only the unambiguous 1:1 case still auto-assigns; multiple unknowns become suggestions in the naming prompt. - Dead code removed: `AppModel.namingDismissTask` (never scheduled), `showNamingPrompt` (auto-open plumbing), unused `NamingCandidateRow` view. -- ⚠️ Written in a Linux container without a Swift toolchain — needs `swift build` + `swift run HeardTests` on macOS CI to verify. +- First pass verified green on macOS CI (run 165: build + full test suite). + +**Speaker labeling follow-up (same branch, second pass):** +- **Speaking Time metric.** `SpeakerProfile.totalSpeakingTime` (backward-compatible decode, defaults 0 for old files) accumulates the sum of each speaker's *pre-merge* transcript segment durations — `mergeConsecutive` extends blocks across intra-speaker silence, so merged segments would overcount. Flows through `UnmatchedSpeaker` → `NamingCandidate` → profile creation/merge, and `SpeakerStore.updateStats` gained an `addSpeaking` field. New sortable "Speaking Time" column in the Speakers table ("Time in Meetings" retained; both use a shared `durationText` formatter: "2h 05m" / "14m" / "38s"). +- **Pending naming candidates survive restart.** `NamingCandidate` is now Codable; `NamingCandidateStore` (`naming_candidates.json`) mirrors every change to `AppModel.namingCandidates` via `didSet` and reloads at bootstrap — dropping clips whose files are gone and whole candidates left clipless (same "can't listen → can't identify" rule as the pipeline). Pending clips are added to the stale-recordings-cleanup preserve set so the 48 h sweep can't strand them. Restored candidates re-raise the `.userAction` badge; `startWatching`/`stopWatching` no longer clobber it. +- **Voice match strictness is user-tunable.** `AppSettings.speakerMatchThreshold` (default 0.30, range 0.15–0.45) feeds `SpeakerMatcher.matchSpeakers(matchThreshold:)`; new Advanced → Voice Matching card (slider "Stricter" ↔ "Looser" + reset), parallel to the Diarization card. The confidence margin (0.10) remains a constant. +- **Roster suggestions are now honest.** Instead of pairing the *i*-th sorted roster name to the *i*-th candidate (arbitrary), every candidate carries the full unmatched-roster list (`NamingCandidate.suggestedNames`) rendered as tappable chips ("maybe: Alice · Bob …", capped at 3 + overflow count) that fill the name field; the field is only pre-filled when exactly one roster name is unmatched (the unambiguous case). Split parts inherit the parent's suggestions. +- **N-way merge.** "Merge Selected" now accepts 2+ profiles. The survivor is picked by `SpeakerMatcher.mergePrimary` (human-given name beats placeholder, then most meetings, then oldest `firstSeen` — unit-tested); one consent dialog covers all explicitly-named profiles being folded in (placeholders rewrite without asking). `SpeakerStore.merge` now also caps merged embeddings at `maxEmbeddingsPerSpeaker` and clips at 5 (overflow clip files deleted) and sums `totalSpeakingTime`. +- Tuple cleanup: the 5-field unmatched-speaker tuple threaded through `TranscriptDocument`/`AudioClipExtractor` is now the `UnmatchedSpeaker` struct. +- New tests: updateStats speaking time, merge caps/speaking sum, NamingCandidateStore round-trip + missing-clip pruning, custom match threshold, mergePrimary ordering. **UI file split (tech debt):** the ~1.9 kLOC `Views.swift` was split by view area into `DesignSystem.swift`, `MenuBarView.swift`, `SpeakerNamingView.swift`, `SettingsView.swift`, `SettingsTabs.swift`, and `SettingsComponents.swift` (all in `Sources/HeardCore/`). Pure move — no logic or symbol changes; `swift build` is clean and all 181 tests pass. @@ -163,13 +172,13 @@ The dictation feature captures mic audio, transcribes in real-time, and injects - Menu bar dropdown (268 px wide) with status card (dark `#2E3338` bg while recording/dictating, Paper bg otherwise), recording timer, and quick actions - Menu bar icons are SF Symbols with symbol effects (`recordingtape`, `record.circle` + `.breathe`, `waveform` + `.variableColor`, `exclamationmark.circle.fill`, `person.crop.circle.badge.exclamationmark`) - **Menu bar reactivity fix**: `MenuBarView` holds direct `@ObservedObject` subscriptions to `queueStore`, `recordingManager`, `pipelineProcessor`, and `meetingDetector` — required because `MenuBarExtra(.window)` does not reliably re-render from forwarded child-store `objectWillChange` events. `MeetingDetector` is now an `ObservableObject` with `@Published isWatching`; `MenuBarIcon` subscribes directly so the paused/dimmed state reflects toggles immediately. `PipelineProcessor.runNextIfNeeded` also recovers orphaned non-terminal jobs (left in mid-stage when the processor is idle) by re-queuing them, charging a retry against the lifetime cap. -- Settings window (880×600, opened via `@Environment(\.openWindow)`) with 6 tabs: **General** (launch at login, auto-watch, developer mode, custom vocabulary, output folder, permissions, meeting notes hotkey), **Transcription** (model download status, pipeline keep-alive, force-unload), **Dictation** (enable, push-to-talk, hotkey recorder, model keep-alive, custom formatting commands, live status), **Speakers** (your name, inline rename, merge, delete, search/sort), **Advanced** (diarization clustering threshold slider with "More speakers" ↔ "Fewer speakers" labels, live numeric readout, reset-to-default), **About** +- Settings window (880×600, opened via `@Environment(\.openWindow)`) with 6 tabs: **General** (launch at login, auto-watch, developer mode, custom vocabulary, output folder, permissions, meeting notes hotkey), **Transcription** (model download status, pipeline keep-alive, force-unload), **Dictation** (enable, push-to-talk, hotkey recorder, model keep-alive, custom formatting commands, live status), **Speakers** (your name, inline rename, merge, delete, search/sort), **Advanced** (diarization clustering threshold slider with "More speakers" ↔ "Fewer speakers" labels, voice match strictness slider with "Stricter" ↔ "Looser" labels, live numeric readouts, reset-to-default), **About** - Standalone "Name Speakers" window scene (id `speaker-naming`, 560×520) with per-candidate audio playback, roster suggestions, and per-candidate Split voices / Discard actions — user-paced, no countdown - Keyboard input works in Settings — `WindowActivationCoordinator` reference-counts `.accessory`/`.regular` transitions across the Settings and Name Speakers windows so closing one while the other is still open never steals keyboard focus - Output folder picker via `NSOpenPanel` - Custom vocabulary management lives in the General tab (add/remove terms, 3-char min, 50-term cap) — terms applied to both transcription and dictation via CTC boosting - Custom formatting commands live in the Dictation tab (map spoken phrases like "new paragraph" to written text like `\n\n`) — applied to both transcription and dictation via ITN rules -- Speaker table with inline rename, merge, delete (context menu), search, and sort (Name / Meetings / Time in Meetings / Last Seen); a leading **Voice** column has a play/stop button that replays the speaker's saved voice clip via a shared `SpeakerClipController` so only one clip plays at a time +- Speaker table with inline rename, N-way merge, delete (context menu), search, and sort (Name / Meetings / Time in Meetings / Speaking Time / Last Seen); a leading **Voice** column has a play/stop button that replays the speaker's saved voice clip via a shared `SpeakerClipController` so only one clip plays at a time - Model download status with progress bars and per-card download buttons, plus a "Download All Models" shortcut - Permission status with grant buttons and System Settings deep-links (Microphone + Screen Recording + Accessibility), surfaced inside the General tab - Launch at login via `SMAppService` @@ -183,11 +192,10 @@ The dictation feature captures mic audio, transcribes in real-time, and injects - Used for automatic speaker name assignment when diarization detects unmatched speakers ### Speaker Naming Prompt (Fully Working) -- Dedicated "Name Speakers" window, **user-initiated** — opened from the menu bar dropdown ("Name Speakers…" row, orange badge) or the Speakers settings tab banner; it does not auto-open and there is no countdown. Candidates stay pending until the user saves, skips, splits, or discards them; closing the window keeps them pending. +- Dedicated "Name Speakers" window, **user-initiated** — opened from the menu bar dropdown ("Name Speakers…" row, orange badge) or the Speakers settings tab banner; it does not auto-open and there is no countdown. Candidates stay pending until the user saves, skips, splits, or discards them; closing the window keeps them pending, and pending candidates persist across app restarts (`naming_candidates.json`). - Each unmatched speaker shows **playable audio clips** (up to 3 × ~10s of their clearest speech from diarization). Speakers with no extractable clip never reach the prompt — a voice the user can't hear can't be identified. - Audio playback via `AVAudioPlayer` with play/stop toggle per speaker -- Suggested names from Teams roster when available (shown as orange hint text) -- Text fields pre-populated with roster suggestions for quick confirmation +- Suggested names from the Teams roster when available: every unmatched roster name is offered on every candidate as tappable chips that fill the name field (no arbitrary name↔voice pre-pairing); the field is pre-populated only when exactly one roster name is unmatched - **"Save & Close"** commits all entered names and dismisses the window via `dismissWindow(id: "speaker-naming")`; **"Skip All"** saves remaining unnamed speakers with `Speaker N` labels and dismisses - **"Split voices"**: when a candidate's samples are different people (diarization merged two voices into one cluster), a per-row button splits it into one sub-candidate per clip, each carrying a clip-local embedding, so each voice can be named separately. **"Discard"** drops a candidate without creating a `SpeakerProfile` (clips deleted immediately; transcript keeps the placeholder label). - Speaker profiles created with voice embeddings from diarization, enabling future recognition. Naming a candidate with an existing profile's name merges into that profile instead of duplicating it.