Skip to content

feat(player): identify songs on Linux from Harbor's own audio - #82

Merged
Thunderhawkk merged 5 commits into
Harborofficial:beta-branchfrom
kalashnikxvxiii:feat/linux-song-id
Nov 6, 2026
Merged

Thunderhawkk merged 5 commits into
Harborofficial:beta-branchfrom
kalashnikxvxiii:feat/linux-song-id

Conversation

@kalashnikxvxiii

@kalashnikxvxiii kalashnikxvxiii commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Song identification now works on Linux. The Identify song button was already visible there, but every press came back with "Song identification is only supported on Windows for now", because the only capture path was the WASAPI loopback.

Linux records through Pulse (which pipewire-pulse also speaks) and attaches to a single sink input: Harbor's own mpv stream, located by the audio-client-name we already set. The rest of the pipeline — silence check, WAV muxing, AudD, Gemini — was already portable and is shared as is.

Two commits:

  1. build(linux) — libpulse-dev in the two workflows that compile natively on Linux.
  2. feat(player) — the Pulse capture path, plus the Cargo.toml/Cargo.lock entry for libpulse-binding.

Why

Picking Harbor's sink input rather than a device loopback is the better fit, not just the portable one: a call or a browser tab playing in the background can never end up in the clip, which the Windows loopback cannot guarantee.

audio-client-name is now a pub(crate) const AUDIO_CLIENT_NAME in mpv.rs, so the string mpv announces and the string song_id looks for cannot drift apart.

As a side effect the Gemini module stops being dead code on Linux, which takes cargo check from 25 warnings to 12.

Verification

  • cargo check --manifest-path src-tauri/Cargo.toml — clean. Warning count drops from 25 on beta-branch to 12, all 12 pre-existing and unrelated.
  • tsc --noEmit -p tsconfig.json — clean.
  • vp fmt --check on the changed files in this PR — passes.
  • Manual, on CachyOS with PipeWire: with a scene playing, pactl list sink-inputs shows application.name = "Harbor" for the mpv stream, distinct from the lowercase "harbor" of the WebView. The case-sensitive match plus the corked check selects the mpv one.

Platform Impact

  • Linux — new behavior, verified on PipeWire via pipewire-pulse. A plain PulseAudio server takes the same path.
  • Windows — unchanged; the WASAPI path is untouched beyond the client-name constant.
  • macOS — unchanged, still returns the unsupported error, now worded "Windows and Linux".
  • TV input — none; the control already existed and was never platform-gated in the UI.

Two follow-ups I deliberately left out of this PR, both to keep it focused:

  • The Gemini setting description still reads "on Windows" (provider-keys.tsx, plus 16 locale catalogs). I had the fix ready, but settings-refinements.ts is unformatted relative to the repo's own convention in every locale, so CI's changed-files vp fmt --check gate turns that one-phrase edit into ~8,300 insertions / 4,200 deletions of pure rewrapping. That reformat deserves its own PR, or a repo-wide pass, rather than riding along with a native feature. Happy to open either.
  • flatpak/cargo-sources.json will need libpulse-binding and libpulse-sys before the next Flatpak build. I did not touch it: the file is already ~300 entries behind beta-branch's own Cargo.lock, so hand-inserting four entries would imply it is current when it isn't — flatpak/update-flatpak.sh regenerates it properly. The Flatpak workflow only runs on pushes to main, so it does not gate this PR.

Checklist

  • This pull request is focused and contains no unrelated refactors.
  • I ran vp check for the changed files.
  • I ran vp run typecheck after TypeScript changes.
  • I ran the relevant Cargo checks after Rust changes.
  • I tested affected platforms when the change is platform-specific.
  • I preserved playback, navigation, and configured hotkey behavior where applicable.
  • I added or updated tests for behavior changes. The capture path needs a live PulseAudio server and a playing mpv stream, so it is covered by the manual check above rather than by a unit test.
  • I removed secrets, tokens, private URLs, and personal data from logs and screenshots.

The song id capture on Linux links libpulse through libpulse-binding,
and libpulse-sys needs the headers and pkg-config file at build time.
Added to both workflows that compile the app natively on Linux: the CI
check and the app build.
The Identify song button showed up on Linux but always failed with
"only supported on Windows for now", because the only capture path was
the WASAPI loopback.

Linux now records through Pulse, which pipewire-pulse also speaks, and
it records a single sink input: Harbor's own mpv stream, found by the
audio-client-name we already set. Unlike the Windows loopback, a call
or a browser tab playing in the background never ends up in the clip.
The rest of the pipeline (silence check, WAV, AudD, Gemini) was already
portable and is shared as is.

The client name moves into a constant so mpv and song_id can't drift
apart. As a side effect the Gemini module stops being dead code on
Linux, which clears 13 warnings.
@kalashnikxvxiii

Copy link
Copy Markdown
Contributor Author

Notes on the force-push and on the red checks, since neither is explained by the description any more.

Why the branch shrank. The first push had a third commit dropping "on Windows" from the Gemini setting description. The format gate runs on changed files only, and settings-refinements.ts is unformatted against the repo's own convention in all 16 locales — so that one-phrase edit dragged in ~8.3k insertions / 4.2k deletions of pure rewrapping, which is why the first run went red at 28s. I dropped the commit instead of landing that here. Happy to send it separately, or to do the locale reformat as its own pass first, whichever you prefer.

Dropping it also surfaced that Cargo.lock had been amended into that commit, so the libpulse-binding entry would have gone missing from the PR. It now sits in the feat commit next to Cargo.toml.

The current red is pre-existing. Both jobs fail on beta-branch as it stands, and this branch reproduces that exactly, not one failure more:

beta-branch (run for #61, since merged as 85699e2) this PR
Frontend 94 distinct failing tests the same 94, identical set
Native 1387 passed; 71 failed; 81 ignored 1387 passed; 71 failed; 81 ignored

The Native failures are all in games::sims_store, games::stardew_store, games::transfers and music::spotify; the Frontend ones are mostly a missing src-tauri/resources/harbor-stream-blocker/README.md, i.e. a setup resource CI never fetches. Nothing near song_id.

Worth noting what the Native job does prove, though: it got through building and running 1458 tests, so libpulse-binding links fine in CI with the libpulse-dev added here. That was the one thing about this change I could not verify locally, since cargo check never links.

@Thunderhawkk

Copy link
Copy Markdown
Contributor

Thanks for the Linux song identification implementation and for verifying it on PipeWire. I reviewed the updated PR, and there are two remaining issues I'd like addressed before merging.

1. [P1] Explicitly connect to Harbor's monitor source

In song_id.rs, the implementation correctly finds Harbor's sink input and calls set_monitor_stream(index). However, connect_record(None, ...) still allows PulseAudio to select the default recording source.

According to PulseAudio's documentation, set_monitor_stream() only isolates a specific sink input when the recording stream is connected to the monitor source of the sink containing that input.

Could you resolve the selected sink input's sink and its corresponding monitor source, then explicitly pass that monitor source to connect_record()?

It would also be helpful to verify the actual recording source and test identification while another application is playing audio, ensuring that only Harbor's playback is captured.

2. [P2] Distinguish between multiple Harbor instances

The current sink-input lookup selects the first uncorked stream with application.name = "Harbor".

If two Harbor instances are playing simultaneously, both could match, potentially causing song identification to capture audio from the wrong instance.

Could you associate the selected sink input with the specific Harbor instance or active mpv playback session? An instance-specific identifier or another reliable stream identity would prevent accidentally selecting another instance's audio.

A test with two Harbor instances playing different audio would help verify this behavior.

Once these two issues are addressed, I'll re-review the updated implementation.

set_monitor_stream() only isolates a sink input when the recording stream
sits on the monitor source of the sink that input plays to. The capture
passed None to connect_record, so the server picked the default source
instead, which on a normal desktop is a capture chain and not a monitor
of anything.

Resolve the selected sink input's sink, ask it for its monitor source and
connect the recording stream to that by name. pipewire-pulse happens to
honour set_monitor_stream whatever source it is given, so this is a no-op
there, but it is what the documented contract asks for and it is what a
plain PulseAudio server needs.
The sink-input lookup took the first uncorked stream announcing the mpv
client name, so two Harbor instances playing at once could each identify
the other's audio.

libmpv plays from inside the Harbor process, so its Pulse client carries
our own pid. Resolve the client behind each candidate sink input and keep
only the ones belonging to this process. As a side effect the WebView,
which registers as "harbor" from a child process, can no longer match
either.

The predicate is a plain function so the selection rules are covered by
tests without a running sound server.
@kalashnikxvxiii

Copy link
Copy Markdown
Contributor Author

Both addressed, one commit each. Thanks for the set_monitor_stream catch — you were right about the contract, and measuring it turned up something worth recording here.

[P1] Explicit monitor source — 1e45d70e's parent resolves the selected sink input's sink, asks it for monitor_source_name, and passes that to connect_record by name. The three introspection queries now share a small drain() helper so each one stays bounded by the same deadline.

I measured the before/after against a module-null-sink with two simultaneous inputs on it — a 440 Hz tone and a silent stream — recording 3 s and comparing RMS, so the isolation claim is falsifiable rather than asserted:

target input connect_record(None) + set_monitor_stream connect_record(monitor) + set_monitor_stream connect_record(monitor), no isolation
tone (loud) 2047.5 2047.5 2032.8
silent, loud sibling on same sink 0.0 0.0 2032.8

The third column being loud on both rows confirms the tone really was on the sink; the two zeroes confirm set_monitor_stream isolates a single input rather than capturing the mix.

One honest correction to my own reasoning, since it changes how this should be read: on pipewire-pulse the old code was not broken. Columns one and two are identical, because the compat layer honours set_monitor_stream whatever source it is handed — it does not fall back to recording the default source. So this is not a user-visible bug fix on PipeWire; it is the code no longer depending on undocumented leniency from one implementation, and it is what a plain PulseAudio server needs per the documented contract. I could not test plain PulseAudio — this machine is PipeWire — so that half is reasoned from the docs, not measured. Worth knowing before you weigh the change.

[P2] Instance identity — rather than inventing an identifier, there is already an exact one: libmpv plays from inside the Harbor process, so the Pulse client behind its sink input carries application.process.id equal to our own pid (verified against pgrep harbor on this box). The lookup now resolves the client behind each candidate and keeps only those belonging to this process, so two instances cannot match each other by construction — different processes, different pids.

It also closes a gap I had not thought about: the WebView registers as harbor, lowercase, from a child process. The old case-sensitive name match already excluded it, but incidentally; now it is excluded on identity too.

The selection rules are a plain is_our_stream() predicate, so they are covered by five unit tests that need no sound server: our own playing stream, a corked stream, another instance's client, other applications and the lowercase WebView, and a stream with no client at all.

Verification

  • cargo check clean; cargo test --lib song_id 17 passed.
  • vp fmt --check on the changed files passes.
  • Full native build run and song identification exercised end to end on CachyOS with PipeWire, with other applications playing at the same time: only Harbor's own playback is recognised.

The two pre-existing follow-ups from my earlier comment still stand unchanged: the "on Windows" string in the Gemini description, and flatpak/cargo-sources.json needing the two new crates.

@kalashnikxvxiii

kalashnikxvxiii commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor Author

Correction to my previous comment, before you re-review: I claimed the P1 commit is "what a plain PulseAudio server needs per the documented contract", and said I could not test that. I have now tested it, and the claim was wrong.

Why I had skipped it. This machine runs PipeWire, and the pulseaudio package carries Conflicts With: pipewire-pulse. Installing the real daemon would have uninstalled pipewire-pulse and taken down audio for the whole desktop until I put it back, so testing the documented path looked like it meant breaking the box, and I left that half of the argument resting on the documentation instead of on a measurement. That was the wrong call to leave standing in a PR you are deciding whether to merge.

How it turned out to be testable after all. The conflict is only about owning the system audio stack. libpulse was already installed at exactly the version the daemon depends on (17.0+r134+g77d25a1e6), so there was no need to install anything: download the package, unpack it into a directory, and run the daemon straight from there on a private UNIX socket, with PULSE_RUNTIME_PATH pointed somewhere private, two null sinks and no hardware modules. It never touches the running PipeWire graph, and clients reach it through PULSE_SERVER. Nothing on this machine was modified.

So: the real PulseAudio 17.0 daemon, with the default source deliberately set to other_sink.monitor — not the monitor of the sink holding the target input. Same 3 s RMS method as before:

target input connect_record(None) + set_monitor_stream connect_record(monitor) + set_monitor_stream connect_record(monitor), no isolation
tone (loud) 2047.5 2047.5 2047.5
silent, loud sibling on same sink 0.0 0.0 2047.5

Identical to PipeWire: the first two columns match on both rows. And the mechanism is visible rather than inferred — while the first case was recording, pactl list source-outputs on that daemon showed the stream attached to Source: 0 = probe_sink.monitor, while the default source was other_sink.monitor (source 1). So real PulseAudio redirects the record stream to the monitor of the sink holding the target input and ignores the device it was given; direct_on_input takes precedence.

So the original code was correct on both server implementations, and P1 fixes no user-visible behaviour on either. What it does buy is that the code no longer depends on that redirect at all: it names the source it wants, which is what the set_monitor_stream documentation describes as the valid configuration. The cost is one extra introspection round trip.

Given that, P1 is your call and I have no stake in keeping it:

  • Keep it if you would rather not rely on redirect behaviour that neither implementation documents.
  • Drop it and I will revert that commit, leaving P2 alone — it is three queries where one used to do, for no measured gain.

Either way P2 stands on its own: the multiple-instance problem was real, and the pid-based client match is unaffected by any of this.

Caveat on the above: one daemon version (17.0) and null sinks, not a hardware capture device as the default source. The source-output evidence suggests the redirect is independent of the source's type, but I have not tried every shape.

Use a stable per-process UUID in the mpv PulseAudio client name and match that exact name during song identification, in addition to the Pulse client PID. Independent Flatpak sandboxes can reuse the same PID; this prevents matching their unrelated playback streams. Keep the Windows and macOS audio-client names unchanged and add same-PID isolation tests.
@Thunderhawkk
Thunderhawkk merged commit 504c4bf into Harborofficial:beta-branch Nov 6, 2026
0 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants