Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions Package.resolved

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

92 changes: 29 additions & 63 deletions ROADMAP.md
Original file line number Diff line number Diff line change
@@ -1,77 +1,43 @@
# Heard — Roadmap

A living list of planned improvements and stretch ideas. Ordered by how close each item is to the current scope of `spec.md`. Anything that would change the product (cloud APIs, LLM integration, non-English transcription, Google Meet support) belongs in [What's explicitly NOT in scope](#whats-explicitly-not-in-scope) unless the spec itself changes.
A living list of known issues, technical debt, and core constraints.

## Near-term — polish & stability
To preserve the focus, performance, and simplicity of Heard as a lean, native, single-process on-device app, we have explicitly pruned speculative enhancements (such as manual speaker splitting, alternative formats, live captions, or visual onboarding wizards) in favor of keeping the core tool highly polished and reliable.

These land inside the existing v1 scope and mostly tighten things the user already sees.
## Technical Debt & Known Issues

### Distribution & install
- ~~**DMG packaging.**~~ Done — `scripts/dmg.sh` builds, signs, notarizes, and packages. v0.1.0 DMG published to GitHub Releases.
- ~~**Homebrew Cask.**~~ Done — `brew tap execsumo/heard && brew install --cask heard` ([tap repo](https://github.com/execsumo/homebrew-heard)).
- ~~**CI publish step.**~~ Done — on tag push, CI builds a release bundle, zips it with `ditto`, and uploads to GitHub Releases via `softprops/action-gh-release`. Notarization is stubbed out (commented step) pending Apple Developer secrets (`APPLE_ID`, `APPLE_APP_PASSWORD`, `APPLE_TEAM_ID`).
- ~~**Update checker.**~~ Done — lightweight GitHub Releases poll on startup (24h interval). Shows banner in menu bar dropdown and Settings → About when a newer version is available; links directly to the GitHub release page. No auto-install — user re-runs the DMG or `brew upgrade --cask heard`.


### Meeting detection & recording
- **Audible meeting-start chime (opt-in).** A short non-intrusive sound confirms recording started. Off by default.

### Pipeline
- ~~**Preprocessing concurrency guard.**~~ Done — "Low Memory Mode" toggle in Settings → Advanced → Memory serializes preprocessing to halve peak RAM (~400 MB vs ~800 MB). Off by default.
- **Per-job log viewer.** When a job fails, the error string is short. Capture a rolling per-job log (stdout/NSLog lines) and show it in a disclosure view.

## Mid-term — within-spec enhancements

Features that fit the on-device, single-process philosophy but require more code than a polish pass.

### Speaker management
- **Manual speaker split.** Inverse of merge — split a speaker profile if the user realizes two voices were collapsed.
- **Split candidate at naming time.** When the naming prompt's clips reveal that diarization merged two voices into one cluster, let the user split the candidate into multiple `SpeakerProfile`s instead of just discarding it. Today the only options are name (and accept the merged embedding) or discard via "Multiple speakers" (no profile created). A future "Split…" action would let the user assign different names per clip and re-cluster the underlying segments — requires segment-level reassignment and a more involved naming UI.
- ✅ **Bulk-delete / archive old speakers.** Speakers with no meeting activity in N months — implemented as automatic deletion at launch based on `speakerRetentionDays` setting (default 90 days), configurable in Advanced → Speaker Archive. Set to 0 to disable.

### Pipeline & output
- **Alternative output formats.** Out of scope per spec today, but worth re-evaluating: plain `.txt`, `.srt`, VTT for video workflows, or a lightweight HTML with anchors.
- **Grounded speaker assignment.** Use the WeSpeaker embedding on the mic track to double-check that the "longest-duration speaker on mic = local user" heuristic holds. If it doesn't, fall back to prompting the user.

### Dictation
- **Dictation transcript log.** Optional history of dictated text (with timestamps and target-app name) for recovery if injection drops characters.

### Design ideas (from `design_handoff_app_surfaces`)
These surfaces are specified in the design handoff but not yet implemented:
- **First-run onboarding flow.** A 620×480px four-step permission wizard (Microphone → Screen Recording → System Audio → Accessibility) with progress indicator and Grant/Skip buttons. Shows on first launch before the main UI.
- **About as a modal sheet.** The design specifies About as a dimmed overlay sheet (380px wide) over the app, rather than a settings tab. Current tab is functional; this would be a polish upgrade.

### Diagnostics
- **In-app diagnostics pane.** Promote `scripts/diagnose.swift` into a hidden developer pane that surfaces Teams process IDs, active power assertions, CoreAudio device tree, and the most recent tap status.
- **One-click bug report bundle.** Zip up recent logs, `pipeline_queue.json`, and (opt-in) the last failed recording's WAVs for manual sharing.

## Long-term — bigger bets

These stretch the architecture and deserve a spec update before landing.

- **Live captions during the meeting.** The spec explicitly disables this to keep the Mac cool during calls; revisit once Apple Silicon idle cost improves or the user can opt-in per-meeting.
- **Sortformer diarizer.** FluidAudio includes `SortformerDiarizer` (~11% DER vs ~17.7% for current LS-EEND + WeSpeaker). Blocked by an embedding gap: Sortformer's `DiarizerTimeline` carries no per-segment speaker embeddings, which the cross-meeting speaker identity system requires. Unblock by adding a WeSpeaker embedding-extraction pass on Sortformer's segments before converting to `DiarizationResult`. Selectable per-meeting size makes sense (Sortformer has 4 fixed speaker slots).
- **Live speaker identification during the meeting.** Would need a streaming diarizer; today's LS-EEND is offline.

## Technical debt
These represent the remaining areas of focus to clean up and stabilize:

### Active Tech Debt & Known Issues
- **In-meeting note editing.** Today the user edits notes by opening the rendered `.md` directly. A future polish: a "Notes" disclosure on each completed job in the menu bar dropdown that lists captured notes and lets the user edit/delete before the transcript is finalized (or rewrite the `.md` if it's already been written).
- **`Views.swift` size.** ~1.9 kLOC for all UI after the Paper design system landed. Split by tab once we're past the early iteration phase.
- **`SlidingWindowAsrConfig` doesn't expose `TdtConfig`.** The internal `asrConfig` hardcodes `TdtConfig()` (blankId 8192 = v3 default). `AsrManager` auto-adapts the blankId when it detects a mismatch against the loaded model, so v2 models work correctly today — but if FluidAudio ever removes that adaptation, v2 dictation would silently decode incorrectly. Bumped to FluidAudio 0.14.5. Upstream fix in review: [FluidInference/FluidAudio#611](https://github.com/FluidInference/FluidAudio/pull/611). Once merged, update `DictationManager.start()` per the TODO comment.

## Non-goals (from `spec.md`)

These are intentional exclusions. Don't add them without a spec update.

- LLM integration (no Claude, OpenAI, or local LLM)
- Cloud APIs of any kind
- **`Views.swift` size.** All SwiftUI UI components live in a single ~1.9 kLOC file after the Paper design system was implemented. Split this by tab or view area to improve build times and maintenance once early UI iteration is finished.
- **Menu bar dropdown height clipping.** The menu bar dropdown uses `.window` style and has a fixed max height. The jobs list can clip when many jobs accumulate.
- **Dictation auto-resume.** Dictation does not auto-resume after a meeting ends (it auto-pauses when a meeting starts to avoid injecting audio, but the user must restart it manually).
- **Teams detection localization.** Teams detection currently matches localized app names; non-English macOS system locales might fail to detect Teams meetings.

### Completed Technical Debt & Polish
- ~~**`SlidingWindowAsrConfig` doesn't expose `TdtConfig`.**~~ Done — FluidAudio updated to 0.14.7. We now pass the explicit `TdtConfig(blankId: modelVersion.blankId)` when initializing `SlidingWindowAsrManager` inside `DictationManager.start()`, avoiding reliance on internal blank-token auto-adaptation.
- ~~**DMG packaging.**~~ Done — `scripts/dmg.sh` builds, signs, notarizes, and packages.
- ~~**Homebrew Cask.**~~ Done — `brew tap execsumo/heard && brew install --cask heard`.
- ~~**CI publish step.**~~ Done — on tag push, CI builds a release bundle, zips it, and uploads to GitHub Releases.
- ~~**Update checker.**~~ Done — lightweight GitHub Releases poll on startup (24h interval). Shows banner in menu bar dropdown and Settings when a newer version is available.
- ~~**Preprocessing concurrency guard.**~~ Done — "Low Memory Mode" toggle in Settings → Advanced → Memory serializes preprocessing to halve peak RAM (~400 MB vs ~800 MB).
- ~~**Bulk-delete / archive old speakers.**~~ Done — automatic deletion at launch based on `speakerRetentionDays` setting (default 90 days), configurable in Advanced → Speaker Archive. Set to 0 to disable.

## Non-goals (from spec.md)

These are intentional exclusions. Do not add them without a formal spec update.

- LLM integration (no Claude, OpenAI, or local LLMs)
- Cloud APIs of any kind (Heard is 100% local)
- Google Meet support (browser-tab; no per-meeting power assertion to detect)
- Manual app recording (pick any app)
- Manual app recording (arbitrary app target selection)
- System-wide audio capture
- Multiple output formats beyond Markdown
- Multiple output formats beyond Markdown (no plain `.txt`, `.srt`, VTT, or HTML)
- Configurable VAD threshold or speaker count
- macOS notifications
- Live captions or live speaker ID during meetings
- Batch import of existing recordings
- Pre-release update channels
- Dictation voice commands (`"scratch that"`, `"new paragraph"`) and spoken punctuation
- Dictation voice commands (`"scratch that"`, `"new paragraph"`) and spoken punctuation (ITN rules handle basic spacing/lines)
- Complex manual speaker splitting or re-clustering tools
6 changes: 2 additions & 4 deletions Sources/HeardCore/DictationManager.swift
Original file line number Diff line number Diff line change
Expand Up @@ -80,10 +80,8 @@ public final class DictationManager: ObservableObject {
// Create a fresh sliding-window manager for this session.
// SlidingWindowAsrManager's input stream is single-use (finish() closes it),
// so a new instance is required each time.
// TODO(FluidInference/FluidAudio#611): use .streaming.applying(tdtConfig: TdtConfig(blankId: modelVersion.blankId))
// once that PR lands. Until then, AsrManager's internal blank-token auto-adaptation
// keeps v2 models working despite the mismatched blankId in SlidingWindowAsrConfig.
let mgr = SlidingWindowAsrManager(config: .streaming)
let tdtConfig = TdtConfig(blankId: modelVersion.blankId)
let mgr = SlidingWindowAsrManager(config: .streaming.applying(tdtConfig: tdtConfig))
try await mgr.loadModels(asrModels!)

// Restore vocab boosting if terms are configured and CTC models are on disk.
Expand Down
87 changes: 57 additions & 30 deletions Sources/HeardCore/Views.swift
Original file line number Diff line number Diff line change
Expand Up @@ -885,7 +885,7 @@ public struct SettingsView: View {
Text("Heard")
.font(.system(size: 13, weight: .semibold))
.foregroundStyle(HeardTheme.Paper.ink)
Text("0.1.0")
Text(model.updateChecker.currentVersion)
.font(.system(size: 10, design: .monospaced))
.foregroundStyle(HeardTheme.Paper.mute)
}
Expand Down Expand Up @@ -971,14 +971,14 @@ public struct SettingsView: View {
sectionGroup("Profile") {
SettingsCard {
CardRow(isLast: true) {
HStack(spacing: HeardTheme.Spacing.sm) {
HStack {
Text("Your Name")
.font(.system(size: 12, weight: .medium))
.foregroundStyle(HeardTheme.Paper.ink)
Spacer()
TextField("Used as speaker label in transcripts", text: settingsBinding(\.userName))
.textFieldStyle(.roundedBorder)
.frame(maxWidth: 220)
.frame(width: 160)
}
}
}
Expand All @@ -999,7 +999,8 @@ public struct SettingsView: View {
}
.labelsHidden()
.controlSize(.small)
.frame(maxWidth: 160)
.frame(maxWidth: .infinity)
.frame(width: 160)
}
}
}
Expand Down Expand Up @@ -1055,7 +1056,8 @@ public struct SettingsView: View {
}
.labelsHidden()
.controlSize(.small)
.frame(maxWidth: 260)
.frame(maxWidth: .infinity)
.frame(width: 160)
}
}
}
Expand Down Expand Up @@ -1094,7 +1096,8 @@ public struct SettingsView: View {
}
.labelsHidden()
.controlSize(.small)
.frame(maxWidth: 260)
.frame(maxWidth: .infinity)
.frame(width: 160)
}
}
}
Expand Down Expand Up @@ -1369,7 +1372,7 @@ public struct SettingsView: View {
.padding(.vertical, 5)
.background(HeardTheme.Paper.surfaceAlt,
in: RoundedRectangle(cornerRadius: HeardTheme.Radius.inline))
.frame(maxWidth: 260)
.frame(width: 160)

Spacer()

Expand Down Expand Up @@ -1435,12 +1438,12 @@ public struct SettingsView: View {
VStack(alignment: .leading, spacing: 4) {
Text("\(readyCount) of \(totalCount) models ready")
.font(.system(size: 15, weight: .semibold))
.foregroundStyle(HeardTheme.Paper.recordingInk)
.foregroundStyle(Color(hex: "F5EFE4"))
Text(model.downloadManager.allBatchModelsReady
? "Ready to transcribe"
: "Some models need downloading")
.font(.system(size: 11))
.foregroundStyle(HeardTheme.Paper.recordingInk.opacity(0.65))
.foregroundStyle(Color(hex: "F5EFE4").opacity(0.65))
}
Spacer()
VStack(alignment: .trailing, spacing: 6) {
Expand Down Expand Up @@ -1613,17 +1616,7 @@ public struct SettingsView: View {
// MARK: About

private var aboutSection: some View {
VStack(spacing: 0) {
LinearGradient(
colors: [Color(hex: "F0E7D5"), Color(hex: "E8DEC8")],
startPoint: .top, endPoint: .bottom
)
.frame(height: 38)
.overlay(alignment: .bottom) {
HeardTheme.Paper.border.frame(height: 0.5)
}

ScrollView {
ScrollView {
VStack(spacing: 0) {
Spacer().frame(height: 40)
HeardMark(size: 72)
Expand Down Expand Up @@ -1685,19 +1678,13 @@ public struct SettingsView: View {
}
.padding(.top, 16)

Text("Powered by FluidAudio · Parakeet TDT · Silero VAD · WeSpeaker")
.font(.system(size: 11))
.foregroundStyle(HeardTheme.Paper.mute)
.padding(.top, 12)

Spacer().frame(height: 40)
}
.frame(minWidth: 500, minHeight: 500)
.background(HeardTheme.Paper.bg)
}
.frame(maxWidth: .infinity, maxHeight: .infinity)
.background(HeardTheme.Paper.bg)
}
.frame(maxWidth: .infinity, maxHeight: .infinity)
.background(HeardTheme.Paper.bg)
}

// MARK: Pane helpers
Expand Down Expand Up @@ -2512,6 +2499,46 @@ struct SpeakerVoiceCell: View {

// MARK: - Flow Layout

private struct WrapLayout: Layout {
var hSpacing: CGFloat = 8
var vSpacing: CGFloat = 8

func sizeThatFits(proposal: ProposedViewSize, subviews: Subviews, cache: inout ()) -> CGSize {
let width = proposal.width ?? .infinity
var x: CGFloat = 0
var y: CGFloat = 0
var rowHeight: CGFloat = 0
for subview in subviews {
let size = subview.sizeThatFits(.unspecified)
if x + size.width > width && x > 0 {
y += rowHeight + vSpacing
x = 0
rowHeight = 0
}
x += size.width + hSpacing
rowHeight = max(rowHeight, size.height)
}
return CGSize(width: width, height: y + rowHeight)
}

func placeSubviews(in bounds: CGRect, proposal: ProposedViewSize, subviews: Subviews, cache: inout ()) {
var x = bounds.minX
var y = bounds.minY
var rowHeight: CGFloat = 0
for subview in subviews {
let size = subview.sizeThatFits(.unspecified)
if x + size.width > bounds.maxX && x > bounds.minX {
y += rowHeight + vSpacing
x = bounds.minX
rowHeight = 0
}
subview.place(at: CGPoint(x: x, y: y), proposal: ProposedViewSize(size))
x += size.width + hSpacing
rowHeight = max(rowHeight, size.height)
}
}
}

struct FlowLayout<Data: RandomAccessCollection, ID: Hashable, Content: View>: View {
private let data: [Data.Element]
private let id: KeyPath<Data.Element, ID>
Expand All @@ -2524,9 +2551,9 @@ struct FlowLayout<Data: RandomAccessCollection, ID: Hashable, Content: View>: Vi
}

var body: some View {
LazyVGrid(columns: [GridItem(.adaptive(minimum: 120), alignment: .leading)],
alignment: .leading, spacing: 8) {
WrapLayout(hSpacing: 8, vSpacing: 8) {
ForEach(data, id: id, content: content)
}
.frame(maxWidth: .infinity, alignment: .leading)
}
}
Loading
Loading