Idea
Detect which tracks from the music library are audible in movies/episodes, and link them with timestamps: a "music in this episode" list on the episode page, tapping through to the track/album — possibly a "now playing" hint during video playback later.
Why the existing segment detection can't do this
The intro/outro detection (disk/.../events/detectsegments/) compares identical recordings of sibling episodes: ChromaFingerprinter produces 32-bit chroma-gradient hashes and SegmentMatcher requires near-exact matches (Hamming ≤ 6) at one global alignment. Music in a show is overlaid with dialogue/SFX, which perturbs the spectrum enough to flip far too many bits — and the problem is a search over the whole music library, not a pairwise comparison within a season.
Proposed approach
Shazam-style landmark fingerprints: spectral peak pairs as hashes. Peaks of prominent music survive a voice-over, and recognition works by voting — each matched hash votes for (track, time offset); a true match shows as a sharp peak in the vote histogram even when only ~20–30% of landmarks survive.
Building blocks to reuse: AudioPcmReader (PCM decode), the FFT in ChromaFingerprinter, the RabbitMQ job pipeline with directory-scoped queues, and the detector-version/sentinel + scanner-backfill idiom from segment detection (segment_detector_version on media_file_entity, MediaFileScanner backfill branch, wipe on re-analysis).
New components:unit tests, same style as ChromaFingerprinter.
2. Library index: fingerprint each music track once, on the node owning the file (directory-scoped event); store hashes in a Postgres table (hash → track id + offset). Because the index lives in the shared database, tracks and videos do not need to share a node — unlike intro detection, which reads sibling files' raw audio locally. (Same trick would incidentally allow cross-node intro detection later: persist fingerprints instead of recomputing from local files.)
3. Detection job per video file: decode the audio in windows on the video's owning node, query the index, vote per (track, offset delta), emit matches above a confidence threshold. CPU cost per movie roughly comparable to one season of intro detection.
4. Schema + API: e.g. media_file_track_segment (media_file_entity_id, track_entity_id, start_in_milliseconds, end_in_milliseconds, confidence), exposed as a field on MediaFile or Episode/Movie via a batch resolver; player selects it and renders the list.
Expectations (be honest in the UI)
- Works well: title songs, needle-drops, credits music — music prominent in the mix.
- Spotty: quiet underscore below loud dialogue.
- Out of scope: covers/different performances than the library's recording (cover-song detection is a different, much harder problem).
Suggested phasing
- Fingerprinter + track indexing + detection job, initially only scanning the credits/title windows (cleanest mix, most visible win; the detected OUTRO segment marks the credits).
- Full-timeline scan + episode/movie "music in this" UI.
- Optional: "now playing" hint during video playback; index-size work for very large music libraries.
Known pitfalls (learned from the segment-detection work)
-
Silent/near-silent audio must be excluded from matching (see the voiced flag added in DETECTOR_VERSION 2) — silence matches everything.
-
Synthetic e2e fixtures need broadband audio (step melodies + seeded pink noise in testdata's create_mkv.sh, Cicada show); sparse pure tones match spuriously.
-
Index size: a few thousand tracks is fine in Postgres (int8 hash, ~thousands of landmarks per track); tens of thousands needs a compaction strategy.
-
E2E would need a testdata fixture where a music-library track's audio is mixed into a video's audio bed, plus the usual pins bump.
Idea
Detect which tracks from the music library are audible in movies/episodes, and link them with timestamps: a "music in this episode" list on the episode page, tapping through to the track/album — possibly a "now playing" hint during video playback later.
Why the existing segment detection can't do this
The intro/outro detection (disk/.../events/detectsegments/) compares identical recordings of sibling episodes: ChromaFingerprinter produces 32-bit chroma-gradient hashes and SegmentMatcher requires near-exact matches (Hamming ≤ 6) at one global alignment. Music in a show is overlaid with dialogue/SFX, which perturbs the spectrum enough to flip far too many bits — and the problem is a search over the whole music library, not a pairwise comparison within a season.
Proposed approach
Shazam-style landmark fingerprints: spectral peak pairs as hashes. Peaks of prominent music survive a voice-over, and recognition works by voting — each matched hash votes for (track, time offset); a true match shows as a sharp peak in the vote histogram even when only ~20–30% of landmarks survive.
Building blocks to reuse: AudioPcmReader (PCM decode), the FFT in ChromaFingerprinter, the RabbitMQ job pipeline with directory-scoped queues, and the detector-version/sentinel + scanner-backfill idiom from segment detection (segment_detector_version on media_file_entity, MediaFileScanner backfill branch, wipe on re-analysis).
Idea
Detect which tracks from the music library are audible in movies/episodes, and link them with timestamps: a "music in this episode" list on the episode page, tapping through to the track/album — possibly a "now playing" hint during video playback later.
Why the existing segment detection can't do this
The intro/outro detection (disk/.../events/detectsegments/) compares identical recordings of sibling episodes: ChromaFingerprinter produces 32-bit chroma-gradient hashes and SegmentMatcher requires near-exact matches (Hamming ≤ 6) at one global alignment. Music in a show is overlaid with dialogue/SFX, which perturbs the spectrum enough to flip far too many bits — and the problem is a search over the whole music library, not a pairwise comparison within a season.
Proposed approach
Shazam-style landmark fingerprints: spectral peak pairs as hashes. Peaks of prominent music survive a voice-over, and recognition works by voting — each matched hash votes for (track, time offset); a true match shows as a sharp peak in the vote histogram even when only ~20–30% of landmarks survive.
Building blocks to reuse: AudioPcmReader (PCM decode), the FFT in ChromaFingerprinter, the RabbitMQ job pipeline with directory-scoped queues, and the detector-version/sentinel + scanner-backfill idiom from segment detection (segment_detector_version on media_file_entity, MediaFileScanner backfill branch, wipe on re-analysis).
New components:unit tests, same style as ChromaFingerprinter.
2. Library index: fingerprint each music track once, on the node owning the file (directory-scoped event); store hashes in a Postgres table (hash → track id + offset). Because the index lives in the shared database, tracks and videos do not need to share a node — unlike intro detection, which reads sibling files' raw audio locally. (Same trick would incidentally allow cross-node intro detection later: persist fingerprints instead of recomputing from local files.)
3. Detection job per video file: decode the audio in windows on the video's owning node, query the index, vote per (track, offset delta), emit matches above a confidence threshold. CPU cost per movie roughly comparable to one season of intro detection.
4. Schema + API: e.g. media_file_track_segment (media_file_entity_id, track_entity_id, start_in_milliseconds, end_in_milliseconds, confidence), exposed as a field on MediaFile or Episode/Movie via a batch resolver; player selects it and renders the list.
Expectations (be honest in the UI)
Suggested phasing
Known pitfalls (learned from the segment-detection work)
Silent/near-silent audio must be excluded from matching (see the voiced flag added in DETECTOR_VERSION 2) — silence matches everything.
Synthetic e2e fixtures need broadband audio (step melodies + seeded pink noise in testdata's create_mkv.sh, Cicada show); sparse pure tones match spuriously.
Index size: a few thousand tracks is fine in Postgres (int8 hash, ~thousands of landmarks per track); tens of thousands needs a compaction strategy.
E2E would need a testdata fixture where a music-library track's audio is mixed into a video's audio bed, plus the usual pins bump.
Idea
Detect which tracks from the music library are audible in movies/episodes, and link them with timestamps: a "music in this episode" list on the episode page, tapping through to the track/album — possibly a "now playing" hint during video playback later.
Why the existing segment detection can't do this
The intro/outro detection (disk/.../events/detectsegments/) compares identical recordings of sibling episodes: ChromaFingerprinter produces 32-bit chroma-gradient hashes and SegmentMatcher requires near-exact matches (Hamming ≤ 6) at one global alignment. Music in a show is overlaid with dialogue/SFX, which perturbs the spectrum enough to flip far too many bits — and the problem is a search over the whole music library, not a pairwise comparison within a season.
Proposed approach
Shazam-style landmark fingerprints: spectral peak pairs as hashes. Peaks of prominent music survive a voice-over, and recognition works by voting — each matched hash votes for (track, time offset); a true match shows as a sharp peak in the vote histogram even when only ~20–30% of landmarks survive.
Building blocks to reuse: AudioPcmReader (PCM decode), the FFT in ChromaFingerprinter, the RabbitMQ job pipeline with directory-scoped queues, and the detector-version/sentinel + scanner-backfill idiom from segment detection (segment_detector_version on media_file_entity, MediaFileScanner backfill branch, wipe on re-analysis).