diff --git a/CHANGELOG.md b/CHANGELOG.md index 1cffdc2..acdf4f2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,42 @@ This is the first documented entry — a snapshot of what SlothArchiver could do as of this release, not a history of every change that got it here. Future releases will log what actually changed from the previous one. +## [1.3.0] — 2026-09-22 +- **Add videos from virtually any yt-dlp-supported site to your library, not + just YouTube.** SoundCloud, Dailymotion, archive.org, PeerTube, and the rest + of the [1800+ sites yt-dlp supports](https://github.com/yt-dlp/yt-dlp/blob/master/supportedsites.md) + can now be added to your permanent library the same way a YouTube video + can — versioned, labelled, and organized by platform instead of by + channel. "Add to library" is no longer greyed out for anything other than + YouTube. + - Every entry plays through the same full-featured player as your YouTube + library — including audio-only sources (SoundCloud, Bandcamp, etc.), + which now show their thumbnail/album art for the whole time they're + playing instead of a blank black screen, and can still be clipped like + any video. + - Each entry shows a color-coded platform label (SoundCloud orange, + Dailymotion blue, Vimeo cyan, Twitch purple, and so on) next to its + title, both on its own page and on its card in the library grid, plus a + new "Non-YouTube" filter to find them all at once. + - A video's page now shows an "Extra data" section with whatever the + source actually provides — uploader, upload date, license, categories, + tags, and, for music platforms, track/artist/album/genre. + - Embed metadata now works for these too, including music tags where + available, and MP3-only sources get their MP3 option folded into + "Convert to" instead of a separate, now-redundant Extract MP3 button. +- **Volume controls in the mini player and queue panel.** Both now have a + volume button that opens a small popover with a vertical slider, so you + can adjust background playback without needing the full player open. +- **Clips now remember their exact start/end timestamps**, and a new + "Mark clip on original video" button jumps back to the source video with + those same timestamps already marked — a quick way to re-clip or adjust + an existing clip without hunting for where it was cut from. +- **Fixed an audio/video sync bug when clipping.** A fast clip could end up + with its audio and video very slightly out of phase — hard to notice on + most content, but obvious on anything music-driven. Clips that need it now + get a quick, targeted re-encode of just the video stream (audio stays a + lossless copy either way) to keep the two in sync. + ## [1.2.0] — 2026-09-18 - **Play anything in the background, and queue up more.** A new mini-player bar at the bottom of the app lets you keep listening (or watching) while diff --git a/README.md b/README.md index 716efc2..7cf5149 100644 --- a/README.md +++ b/README.md @@ -20,7 +20,8 @@ Docker, servers, or a complicated setup to get one.** *Free and open source: no subscriptions, no accounts, no server to run.* SlothArchiver is a free desktop app for downloading and organizing videos and audio -from YouTube and other platforms into a permanent library on your own computer. +from YouTube and virtually any other platform yt-dlp supports into a permanent +library on your own computer. Install it, point it at a folder, and it does the archiving for you: no subscriptions, no re-uploading to yet another cloud service, no losing access when a video gets taken down or a channel disappears. The name isn't ironic: @@ -29,17 +30,17 @@ itself. ## Download - | Platform | Link | |---|---| -| Windows (installer) | [Download](https://github.com/SlothSoftworks/Sloth-Archiver/releases/download/v1.2.0/SlothArchiver.Setup.1.2.0.exe) | -| Windows (portable, no install) | [Download](https://github.com/SlothSoftworks/Sloth-Archiver/releases/download/v1.2.0/SlothArchiver.1.2.0.exe) | -| macOS (Apple Silicon) | [Download](https://github.com/SlothSoftworks/Sloth-Archiver/releases/download/v1.2.0/SlothArchiver-1.2.0-arm64.dmg) | -| macOS (Intel) | [Download](https://github.com/SlothSoftworks/Sloth-Archiver/releases/download/v1.2.0/SlothArchiver-1.2.0.dmg) | -| Linux (AppImage) | [Download](https://github.com/SlothSoftworks/Sloth-Archiver/releases/download/v1.2.0/SlothArchiver-1.2.0.AppImage) | +| Windows (installer) | [Download](https://github.com/SlothSoftworks/Sloth-Archiver/releases/download/v1.3.0/SlothArchiver.Setup.1.3.0.exe) | +| Windows (portable, no install) | [Download](https://github.com/SlothSoftworks/Sloth-Archiver/releases/download/v1.3.0/SlothArchiver.1.3.0.exe) | +| macOS (Apple Silicon) | [Download](https://github.com/SlothSoftworks/Sloth-Archiver/releases/download/v1.3.0/SlothArchiver-1.3.0-arm64.dmg) | +| macOS (Intel) | [Download](https://github.com/SlothSoftworks/Sloth-Archiver/releases/download/v1.3.0/SlothArchiver-1.3.0.dmg) | +| Linux (AppImage) | [Download](https://github.com/SlothSoftworks/Sloth-Archiver/releases/download/v1.3.0/SlothArchiver-1.3.0.AppImage) | All builds are unsigned, so your OS will show a first-run security warning. See [Installing](#installing) below. @@ -93,10 +94,12 @@ That's the whole point of SlothArchiver: a simple catalogued library, organized - **Download video or audio** from YouTube and virtually any other site yt-dlp supports (1800+ platforms, including SoundCloud, TikTok, Instagram, Facebook, Dailymotion, and more) in the quality you choose. -- **A real library, not just a downloads folder.** Every video is organized by - channel, searchable, and keeps a history of versions if you ever re-fetch it. - Most downloaders stop at "file saved somewhere"; SlothArchiver actually keeps - track of what you have. +- **A real library, not just a downloads folder.** Works with YouTube and + virtually any other yt-dlp-supported platform — YouTube videos are + organized by channel, everything else by platform, and either way it's + searchable, filterable, and keeps a history of versions if you ever + re-fetch it. Most downloaders stop at "file saved somewhere"; SlothArchiver + actually keeps track of what you have. - **Playlist archiving**: save an entire playlist at once, and refresh it later to pick up new additions without losing what you already have, with removed videos automatically flagged instead of silently disappearing. @@ -219,7 +222,7 @@ SlothArchiver is a personal-use, local tool: everything it downloads stays in yo >TL;DR: It's a gray area as far as YouTube is concerned but not illegal. This is a personal, local-only archiving tool; staying within your platform's ToS and your local copyright law is on you. #### What platforms does SlothArchiver support downloading from? -At time of writing we support YouTube for downloads, library, and playlists. For pure downloads, we support virtually any site [yt-dlp itself supports](https://github.com/yt-dlp/yt-dlp/blob/master/supportedsites.md) (1800+ platforms), including TikTok, Instagram, Twitter, Facebook, SoundCloud, and Dailymotion, to name a few with dedicated recognition in the app's UI. +Downloads and the library both work with virtually any site [yt-dlp itself supports](https://github.com/yt-dlp/yt-dlp/blob/master/supportedsites.md) (1800+ platforms), including YouTube, TikTok, Instagram, Twitter, Facebook, SoundCloud, and Dailymotion, to name a few with dedicated recognition in the app's UI. Playlists are currently YouTube-only (see "Can I download an entire playlist?" below). #### What operating systems does SlothArchiver run on? Currently we support Mac, Linux, and Windows. This is a solo dev operation and I manually create the executables per system, so I expect there could be OS issues as more people try out the software on their systems. If you experience any issues, let me know by raising an issue and detailing it there. diff --git a/ROADMAP.md b/ROADMAP.md index b1a2aab..87c28d1 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -12,7 +12,6 @@ What's being developed right now? These are some of the topic that I'm working o - **Custom player improvements** - new playback features and general improvements to the app's built-in player. - **More Library flexibility** - More options to tag and explore libraries in different partitions so you can save and catalogue your downloads in different library navigation views if you choose. -- **Expanded library** - Expanding the library function from only youtube to more platforms and adding more flexibility so all your videos can use the built in ffmpeg tools Got a feature you'd like to see? Check the README's FAQ for how to raise it as an issue. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 17a085e..c3eaa62 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -1,13 +1,12 @@ # sloth-archiver — Architecture Overview sloth-archiver is a desktop application (built on Electron, with a React/TypeScript -interface) for downloading and archiving video and audio content — primarily from -YouTube, with download support for virtually any other site yt-dlp itself -supports (1800+ platforms) — into a personal, local library on the user's own -machine. It wraps the well-known `yt-dlp` tool for the actual -extraction/download work and `ffmpeg` for local media processing, and adds a -persistent, browsable library on top: versioned entries, playlists, search, and a -built-in player. +interface) for downloading and archiving video and audio content from YouTube +and virtually any other site yt-dlp itself supports (1800+ platforms) into a +personal, local library on the user's own machine. It wraps the well-known +`yt-dlp` tool for the actual extraction/download work and `ffmpeg` for local +media processing, and adds a persistent, browsable library on top: versioned +entries, playlists (YouTube only, for now), search, and a built-in player. This document explains how the pieces fit together and how data moves through the system for its main features. It is written for a technical reader who wants to diff --git a/package-lock.json b/package-lock.json index 3db9f63..00360da 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "sloth-archiver", - "version": "1.2.0", + "version": "1.3.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "sloth-archiver", - "version": "1.2.0", + "version": "1.3.0", "license": "GPL-3.0-or-later", "dependencies": { "@emotion/react": "^11.14.0", diff --git a/package.json b/package.json index d68454a..1926d2e 100644 --- a/package.json +++ b/package.json @@ -1,7 +1,7 @@ { "name": "sloth-archiver", "private": true, - "version": "1.2.0", + "version": "1.3.0", "description": "A desktop app for downloading and archiving videos, with a built-in library and playlist tracking.", "author": "Sloth ", "license": "GPL-3.0-or-later", diff --git a/src/electron/__fixtures__/ytdlp-info/archive_org.json b/src/electron/__fixtures__/ytdlp-info/archive_org.json new file mode 100644 index 0000000..9d56c8d --- /dev/null +++ b/src/electron/__fixtures__/ytdlp-info/archive_org.json @@ -0,0 +1,8 @@ +{ + "id": "BigBuckBunny_124", + "title": "Big Buck Bunny", + "extractor_key": "ArchiveOrg", + "license": "http://creativecommons.org/licenses/by/3.0/", + "timestamp": 1309554623, + "track": "big buck bunny 720p surround" +} diff --git a/src/electron/__fixtures__/ytdlp-info/dailymotion.json b/src/electron/__fixtures__/ytdlp-info/dailymotion.json new file mode 100644 index 0000000..f0e1ecd --- /dev/null +++ b/src/electron/__fixtures__/ytdlp-info/dailymotion.json @@ -0,0 +1,8 @@ +{ + "id": "xbafapm", + "title": "\"Tokidoki Bosotto Russia-go de Dereru Tonari no Aalya-san (Alya Sometimes Hides Her Feelings in Russian)\" upcoming TV anime season 2 PV", + "extractor_key": "Dailymotion", + "timestamp": 1789872906, + "uploader_id": "x1dagit", + "tags": [] +} diff --git a/src/electron/__fixtures__/ytdlp-info/peertube.json b/src/electron/__fixtures__/ytdlp-info/peertube.json new file mode 100644 index 0000000..c89b00a --- /dev/null +++ b/src/electron/__fixtures__/ytdlp-info/peertube.json @@ -0,0 +1,10 @@ +{ + "id": "9c9de5e8-0a1e-484a-b099-e80766180a6d", + "title": "What is PeerTube?", + "extractor_key": "PeerTube", + "license": "Attribution - Share Alike", + "timestamp": 1538391166, + "uploader_id": "3", + "categories": ["Science & Technology"], + "tags": ["framasoft", "peertube"] +} diff --git a/src/electron/__fixtures__/ytdlp-info/soundcloud.json b/src/electron/__fixtures__/ytdlp-info/soundcloud.json new file mode 100644 index 0000000..43db09b --- /dev/null +++ b/src/electron/__fixtures__/ytdlp-info/soundcloud.json @@ -0,0 +1,11 @@ +{ + "id": "21792829", + "title": "First Of The Year (Equinox)", + "extractor_key": "Soundcloud", + "license": "all-rights-reserved", + "timestamp": 1314199241, + "track": "First Of The Year (Equinox)", + "artist": "Skrillex", + "genre": "Dubstep", + "uploader_id": "856062" +} diff --git a/src/electron/__fixtures__/ytdlp-info/youtube.json b/src/electron/__fixtures__/ytdlp-info/youtube.json new file mode 100644 index 0000000..377f19e --- /dev/null +++ b/src/electron/__fixtures__/ytdlp-info/youtube.json @@ -0,0 +1,10 @@ +{ + "id": "aqz-KE-bpKQ", + "title": "Big Buck Bunny 60fps 4K - Official Blender Foundation Short Film", + "extractor_key": "Youtube", + "license": "Creative Commons Attribution license (reuse allowed)", + "timestamp": 1415628355, + "uploader_id": "@BlenderOfficial", + "categories": ["Film & Animation"], + "tags": ["blender", "animation", "4K", "UHD", "Big Buck Bunny (Film)", "Blender Foundation (Nonprofit Organization)", "Short Film (Film Genre)", "4K Resolution"] +} diff --git a/src/electron/ffmpegUtils.mjs b/src/electron/ffmpegUtils.mjs index 284969a..d63739c 100644 --- a/src/electron/ffmpegUtils.mjs +++ b/src/electron/ffmpegUtils.mjs @@ -280,37 +280,87 @@ export function createFfmpegRunner({ ffmpegBinaryPath, ffprobeBinaryPath, onLog, }); } - // Below this fraction of the clip's own length, or this many seconds in - // absolute terms, a keyframe-rounded start is a rounding error nobody - // will notice (a 2s-early start on a 10-minute clip is 0.3% of it). - // Above it, a real, perceptible chunk of what the user asked for would - // be missing (that same 2s on a 3-second clip is 65% of it) -- these - // numbers aren't tuned against real user reports, just chosen to - // separate those two cases by a wide margin. - const KEYFRAME_RISK_RATIO_THRESHOLD = 0.1; - const KEYFRAME_RISK_ABSOLUTE_FLOOR_SECONDS = 0.5; + // Two independent reasons a copy-mode clip's keyframe-rounded start can + // be wrong -- each gets its own threshold rather than folding them into + // one number, since they don't scale the same way: + // - A/V sync: under stream copy, video can only start on a keyframe, + // but audio has no such restriction and lands almost exactly on the + // requested point -- confirmed directly against a real clip whose + // video and audio, after copy-mode trimming, started ~0.46s apart + // even though the keyframe offset causing it was under 1% of that + // clip's own length (see clipAndConvert's own comment). That gap is a + // perceptual constant, not something that gets less noticeable on a + // longer clip -- ITU-R BT.1359 puts the detectability threshold + // around 45ms (audio ahead of video) to 125ms (audio behind); 100ms + // is a reasonable single cutoff given this is already a coarse proxy + // (measuring video's own rounding, not the actual video/audio + // difference -- audio's rounding is comparatively negligible). + // - Content loss: even a sync-safe offset can still eat a real chunk of + // a *very short* clip (a 0.4s-early start is a rounding error on a + // 10-minute clip, but is most of a 3-second one) -- kept as a + // fraction-of-clip-length check alongside the fixed sync floor above, + // not instead of it, since the two failure modes are unrelated. + const AV_SYNC_RISK_THRESHOLD_SECONDS = 0.1; + const CONTENT_LOSS_RISK_RATIO_THRESHOLD = 0.1; async function isKeyframeRoundingRisky(inputPath, startSeconds, totalDurationSeconds) { if (!(totalDurationSeconds > 0)) return false; const keyframeBefore = await findLastKeyframeAtOrBefore(inputPath, startSeconds); const offset = Math.max(0, startSeconds - keyframeBefore); - return offset > KEYFRAME_RISK_ABSOLUTE_FLOOR_SECONDS - && (offset / totalDurationSeconds) > KEYFRAME_RISK_RATIO_THRESHOLD; + return offset > AV_SYNC_RISK_THRESHOLD_SECONDS + || (offset / totalDurationSeconds) > CONTENT_LOSS_RISK_RATIO_THRESHOLD; + } + + // How much earlier than the requested start to land the fast, + // index-based input seek before handing the small remainder off to an + // exact output-side seek -- see buildSplitSeekArgs's own comment for why + // splitting the seek this way matters. Comfortably larger than the + // keyframe intervals seen in real library files (~2-10s, per + // KEYFRAME_SEARCH_WINDOW_SECONDS's own comment), so the coarse seek + // always lands before any keyframe the fine seek might need to decode + // through, while still keeping the input seek itself a genuine O(1) + // index jump rather than a slow linear read from the start of the file. + const COARSE_SEEK_BUFFER_SECONDS = 15; + + // Splits a single requested start time into a fast, approximate + // input-side seek plus an exact output-side remainder -- the standard + // ffmpeg technique for frame-accurate seeking without paying for a full + // linear read from the start of the file + // (https://trac.ffmpeg.org/wiki/Seeking). Used only for the video-only + // re-encode path below: a single input-side -ss leaves the re-encoded + // video stream frame-accurate regardless (decoding always discards + // frames before the requested point), but leaves the *copied* audio + // stream still landing wherever that one input seek happened to put it + // -- an output-side seek is what actually trims a copied stream + // precisely, packet by packet, without decoding it. See clipAndConvert's + // own comment for the full reasoning and the real numbers behind it. + function buildSplitSeekArgs(startSeconds) { + const coarseSeekSeconds = Math.max(0, startSeconds - COARSE_SEEK_BUFFER_SECONDS); + const fineSeekSeconds = startSeconds - coarseSeekSeconds; + return { + preInputArgs: coarseSeekSeconds > 0 ? ['-ss', String(coarseSeekSeconds)] : [], + fineSeekArgs: fineSeekSeconds > 0 ? ['-ss', String(fineSeekSeconds)] : [], + }; } // "Same as source" (format 'source') means don't change the codec, only - // stop stream-copying -- so when a keyframe-risk re-encode is needed - // there, it still has to re-encode into *something* resembling the - // source rather than a fixed target. Only VP8/VP9 (-> WebM's own codecs) - // is special-cased, matching convertWithFallback/the reencodeCodecArgs - // below -- anything else (including codecs this bundled ffmpeg can't - // encode) falls back to the same libx264/aac default those use, which is - // already correct for the overwhelmingly common case (yt-dlp downloads - // are forced to --merge-output-format mp4, i.e. already h264/aac). - async function reencodeCodecArgsMatchingSource(inputPath) { + // stop stream-copying the video -- so when a keyframe-risk re-encode is + // needed there, it still has to re-encode into *something* resembling + // the source rather than a fixed target. Only VP8/VP9 (-> WebM's own + // codecs) is special-cased, matching convertWithFallback/the + // reencodeCodecArgs below -- anything else (including codecs this + // bundled ffmpeg can't encode) falls back to the same libx264 default + // those use, which is already correct for the overwhelmingly common case + // (yt-dlp downloads are forced to --merge-output-format mp4, i.e. + // already h264). Video-only, not video+audio: audio is never actually at + // risk from keyframe rounding (see isKeyframeRoundingRisky's own + // comment) -- clipAndConvert keeps stream-copying it even on this + // "risky" path, so re-encoding it too would only cost quality/time for + // no sync benefit. + async function reencodeVideoCodecArgsMatchingSource(inputPath) { const streams = await probeMediaStreams(inputPath).catch(() => []); const isVp8or9 = streams.some((s) => s.codecType === 'video' && /^vp[89]$/.test(s.codecName || '')); - return isVp8or9 ? ['-c:v', 'libvpx-vp9', '-c:a', 'libopus'] : ['-c:v', 'libx264', '-c:a', 'aac']; + return isVp8or9 ? ['-c:v', 'libvpx-vp9'] : ['-c:v', 'libx264']; } // Clip [start,end] and, optionally, convert format in one pass -- not a @@ -334,15 +384,34 @@ export function createFfmpegRunner({ ffmpegBinaryPath, ffprobeBinaryPath, onLog, // makes the demuxer jump to the keyframe *at or before* the requested // point, so video and audio both start together at that same boundary -- // zero re-encoding, zero quality/frame loss, just a clip that may start - // up to one GOP length earlier than the exact requested timestamp (the - // standard, universally-recommended tradeoff for lossless trimming). + // up to one GOP length earlier than the exact requested timestamp. + // + // That tradeoff has a real, separate cost of its own, though: audio + // isn't keyframe-bound the way video is, so its own copy-mode trim lands + // much closer to the requested point than video's does -- confirmed + // directly against a real clip (start 00:00:23.290 into an AV1/Opus + // source) where the nearest keyframe at or before that point was at + // 21.855s, and the actual output file's first video packet landed at + // -1.468s relative to the intended start while its first audio packet + // landed at -1.009s -- a ~0.46s *relative* offset between the two + // streams baked permanently into the file, not something a player can + // ever correct for. isKeyframeRoundingRisky below exists specifically to + // catch this -- when it does, the branches below re-encode video only + // (frame-accurate regardless of seek placement, since decoding always + // discards frames before the requested point) while still + // stream-copying audio via buildSplitSeekArgs's output-side fine seek + // (confirmed against the same real file: video and audio then land + // 0.033s/0.011s after the intended start, a ~0.02s difference -- well + // under the ~45-125ms perceptibility range cited in + // isKeyframeRoundingRisky's own comment), rather than the coarse + // single-input-seek used by the fast copy-both path below (which would + // leave audio just as unfixed as video was). // startSeconds (plain seconds, separate from the HH:MM:SS `start` string // ffmpeg itself takes) drives isKeyframeRoundingRisky below -- when that - // tradeoff would actually cost a noticeable chunk of a short clip, this - // skips straight to a real re-encode (frame-accurate, since re-encoding - // decodes every frame rather than copying packets) instead of accepting - // it, without paying that re-encode cost on the vast majority of clips - // where the rounding error is negligible. + // tradeoff would actually cost a noticeable chunk of a short clip, or + // risk an audible A/V sync gap, this skips straight to that video-only + // re-encode instead of accepting it, without paying any re-encode cost + // on the vast majority of clips where the rounding error is negligible. // -to (an absolute output timestamp) is replaced with -t (a duration): // once the input has been seeked, -to's "absolute timestamp" meaning is // no longer relative to the original file, but -t's plain duration is @@ -370,8 +439,24 @@ export function createFfmpegRunner({ ffmpegBinaryPath, ffprobeBinaryPath, onLog, if (isSourceFormat) { if (forceReencode || risky) { - const sourceReencodeCodecArgs = await reencodeCodecArgsMatchingSource(inputPath); - await runFfmpegWithProgress({ inputPath, outputPath, codecArgs: [...durationArgs, ...sourceReencodeCodecArgs], totalDurationSeconds, onProgress, preInputArgs }); + const videoCodecArgs = await reencodeVideoCodecArgsMatchingSource(inputPath); + const codecArgs = [...durationArgs, ...videoCodecArgs, '-c:a', 'copy']; + // The split seek needs a concrete startSeconds to divide -- + // an older/backward-compatible caller that only ever passes + // forceReencode without it (startSeconds == null skips the + // risk check above entirely, but forceReencode can still + // reach here on its own) falls back to the plain single + // input-side seek instead, same placement as the fast path + // below, since there's nothing to split. + if (startSeconds != null) { + const { preInputArgs: splitPreInputArgs, fineSeekArgs } = buildSplitSeekArgs(startSeconds); + await runFfmpegWithProgress({ + inputPath, outputPath, codecArgs, totalDurationSeconds, onProgress, + preInputArgs: splitPreInputArgs, extraInputArgs: fineSeekArgs, + }); + return; + } + await runFfmpegWithProgress({ inputPath, outputPath, codecArgs, totalDurationSeconds, onProgress, preInputArgs }); return; } await runFfmpegWithProgress({ inputPath, outputPath, codecArgs: [...durationArgs, '-c', 'copy'], totalDurationSeconds, onProgress, preInputArgs }); diff --git a/src/electron/ffmpegUtils.test.mjs b/src/electron/ffmpegUtils.test.mjs index 8167ec6..4452eec 100644 --- a/src/electron/ffmpegUtils.test.mjs +++ b/src/electron/ffmpegUtils.test.mjs @@ -61,10 +61,11 @@ beforeEach(() => { }); describe('clipAndConvert', () => { - it('stream-copies (fast path) when the keyframe rounding error is negligible relative to the clip length', async () => { - // 10-minute clip, nearest keyframe just 2s before the requested start -- - // 2/600 is nowhere near either risk threshold. - stubSpawn({ keyframeTimes: [598], streams: [{ codecType: 'video', codecName: 'h264' }] }); + it('stream-copies (fast path) when the keyframe rounding error is negligible for both A/V sync and content loss', async () => { + // 10-minute clip, nearest keyframe just 0.05s before the requested start + // -- under the fixed A/V-sync threshold, and 0.05/600 is nowhere near + // the content-loss ratio threshold either. + stubSpawn({ keyframeTimes: [599.95], streams: [{ codecType: 'video', codecName: 'h264' }] }); const { clipAndConvert } = createFfmpegRunner({ ffmpegBinaryPath: FFMPEG_BIN, ffprobeBinaryPath: FFPROBE_BIN }); await clipAndConvert({ @@ -76,9 +77,10 @@ describe('clipAndConvert', () => { expect(ffmpegCallArgs()).not.toEqual(expect.arrayContaining(['-c:v'])); }); - it('auto-upgrades to a re-encode when the keyframe rounding error would eat a large fraction of a short clip', async () => { + it('auto-upgrades to a video-only re-encode (audio still copied) when the keyframe rounding error would eat a large fraction of a short clip', async () => { // 12-second clip, nearest keyframe 8s before the requested start -- 8/12 - // is a large majority of the clip. + // is a large majority of the clip (and also well past the fixed A/V-sync + // threshold on its own). stubSpawn({ keyframeTimes: [2], streams: [{ codecType: 'video', codecName: 'h264' }] }); const { clipAndConvert } = createFfmpegRunner({ ffmpegBinaryPath: FFMPEG_BIN, ffprobeBinaryPath: FFPROBE_BIN }); @@ -87,11 +89,23 @@ describe('clipAndConvert', () => { format: 'source', totalDurationSeconds: 12, }); - expect(ffmpegCallArgs()).toEqual(expect.arrayContaining(['-c:v', 'libx264', '-c:a', 'aac'])); - expect(ffmpegCallArgs()).not.toEqual(expect.arrayContaining(['-c', 'copy'])); + // Audio is never actually at risk from keyframe rounding (see + // isKeyframeRoundingRisky's own comment), so only video gets re-encoded + // -- audio stays a stream copy, made accurate via the fine output-side + // seek rather than by decoding it. + expect(ffmpegCallArgs()).toEqual(expect.arrayContaining(['-c:v', 'libx264', '-c:a', 'copy'])); + expect(ffmpegCallArgs()).not.toEqual(expect.arrayContaining(['-c:a', 'aac'])); + // Split-seek: a coarse input-side -ss (10 - 15 clamped to 0, i.e. no + // input seek needed here since the buffer already covers the whole + // clip) plus an exact fine seek of the remainder, placed as an output + // option (after -i). + const args = ffmpegCallArgs(); + const iIndex = args.indexOf('-i'); + expect(args.slice(0, iIndex)).not.toEqual(expect.arrayContaining(['-ss'])); + expect(args.slice(iIndex)).toEqual(expect.arrayContaining(['-ss', '10'])); }); - it('matches the source codec (not a fixed default) when auto-upgrading a "same as source" VP9 clip', async () => { + it('matches the source video codec (not a fixed default) when auto-upgrading a "same as source" VP9 clip, still copying audio', async () => { stubSpawn({ keyframeTimes: [2], streams: [{ codecType: 'video', codecName: 'vp9' }, { codecType: 'audio', codecName: 'opus' }] }); const { clipAndConvert } = createFfmpegRunner({ ffmpegBinaryPath: FFMPEG_BIN, ffprobeBinaryPath: FFPROBE_BIN }); @@ -100,7 +114,26 @@ describe('clipAndConvert', () => { format: 'source', totalDurationSeconds: 12, }); - expect(ffmpegCallArgs()).toEqual(expect.arrayContaining(['-c:v', 'libvpx-vp9', '-c:a', 'libopus'])); + expect(ffmpegCallArgs()).toEqual(expect.arrayContaining(['-c:v', 'libvpx-vp9', '-c:a', 'copy'])); + }); + + it('splits the seek across a coarse input-side seek and an exact output-side seek when the requested start is past the coarse-seek buffer', async () => { + // 40-minute-in clip, keyframe risk tripped by a 1s offset alone -- + // exercises the non-zero coarse-seek branch of buildSplitSeekArgs (2400 - + // 15 = 2385 coarse, 15 fine), unlike the other risky-path test above + // where startSeconds is too small for the buffer to matter. + stubSpawn({ keyframeTimes: [2399], streams: [{ codecType: 'video', codecName: 'h264' }] }); + const { clipAndConvert } = createFfmpegRunner({ ffmpegBinaryPath: FFMPEG_BIN, ffprobeBinaryPath: FFPROBE_BIN }); + + await clipAndConvert({ + inputPath: '/in.mp4', outputPath: '/out.mp4', start: '00:40:00', startSeconds: 2400, + format: 'source', totalDurationSeconds: 60, + }); + + const args = ffmpegCallArgs(); + const iIndex = args.indexOf('-i'); + expect(args.slice(0, iIndex)).toEqual(expect.arrayContaining(['-ss', '2385'])); + expect(args.slice(iIndex)).toEqual(expect.arrayContaining(['-ss', '15'])); }); it('goes straight to the format-specific re-encode (skipping the copy attempt) when auto-upgrading a format conversion', async () => { @@ -117,7 +150,7 @@ describe('clipAndConvert', () => { expect(ffmpegCalls[0][1]).toEqual(expect.arrayContaining(['-c:v', 'libvpx-vp9', '-c:a', 'libopus'])); }); - it('never probes for keyframe risk when forceReencode is already set', async () => { + it('never probes for keyframe risk when forceReencode is already set, and still only re-encodes video', async () => { stubSpawn({ keyframeTimes: [598], streams: [{ codecType: 'video', codecName: 'h264' }] }); const { clipAndConvert } = createFfmpegRunner({ ffmpegBinaryPath: FFMPEG_BIN, ffprobeBinaryPath: FFPROBE_BIN }); @@ -127,7 +160,7 @@ describe('clipAndConvert', () => { }); expect(spawnMock.mock.calls.some(([bin, args]) => bin === FFPROBE_BIN && args.includes('-read_intervals'))).toBe(false); - expect(ffmpegCallArgs()).toEqual(expect.arrayContaining(['-c:v', 'libx264', '-c:a', 'aac'])); + expect(ffmpegCallArgs()).toEqual(expect.arrayContaining(['-c:v', 'libx264', '-c:a', 'copy'])); }); it('skips the risk check entirely when startSeconds is not provided (backward-compatible callers)', async () => { @@ -155,7 +188,7 @@ describe('clipAndConvert', () => { // immediately before the requested point (a real seek, not a scan from // zero) using packet-level flags instead of frame-level metadata. it('bounds the keyframe probe to a fixed window before the requested point, not a scan from the start of the file', async () => { - stubSpawn({ keyframeTimes: [3536.5], streams: [{ codecType: 'video', codecName: 'av1' }] }); + stubSpawn({ keyframeTimes: [3536.95], streams: [{ codecType: 'video', codecName: 'av1' }] }); const { clipAndConvert } = createFfmpegRunner({ ffmpegBinaryPath: FFMPEG_BIN, ffprobeBinaryPath: FFPROBE_BIN }); // An hour into a long file -- the exact shape of the real bug report. @@ -178,8 +211,9 @@ describe('clipAndConvert', () => { const [intervalStart, intervalEnd] = interval.split('%').map(Number); expect(intervalEnd).toBe(3537); expect(intervalStart).toBeGreaterThan(3000); - // Negligible real offset (3537 - 3536.5 = 0.5s) -- stays on the fast copy - // path, exactly as it should once the probe itself is fast and correct. + // Negligible real offset (3537 - 3536.95 = 0.05s) -- stays on the fast + // copy path, exactly as it should once the probe itself is fast and + // correct. expect(ffmpegCallArgs()).toEqual(expect.arrayContaining(['-c', 'copy'])); }); diff --git a/src/electron/library.mjs b/src/electron/library.mjs index 6bdc192..b828e79 100644 --- a/src/electron/library.mjs +++ b/src/electron/library.mjs @@ -26,6 +26,12 @@ export const PLAYLISTS_DIR_NAME = 'playlists'; // root). export const CLIPS_DIR_NAME = 'clips'; +// Top-level reserved folder name for non-YouTube library entries (see +// SlothArchiver-dossier plan for non-YouTube platform support), a sibling of +// real per-uploader channel folders -- same reserved-name pattern as +// PLAYLISTS_DIR_NAME/CLIPS_DIR_NAME above. +export const NONYT_DIR_NAME = 'NonYT'; + // SubLibrary / tag library feature (see // SlothArchiver-dossier/futureSpecsFeedback.md's decided design): every tag, // including the default untagged case, is a same-filesystem subfolder @@ -271,7 +277,7 @@ export function transferVideoTags(libraryDir, sourceTag, targetTag, videoIds) { // today" and surface a "this entry predates newer features, refresh it" // notice -- see LibraryVideoDetail.tsx/PlaylistsSection.tsx's own duplicated // copy of these two numbers. -export const CURRENT_VIDEO_SCHEMA_VERSION = 4; +export const CURRENT_VIDEO_SCHEMA_VERSION = 5; export const CURRENT_PLAYLIST_SCHEMA_VERSION = 2; // One cross-platform sanitizer using Windows' illegal-character set as the @@ -324,11 +330,27 @@ export function videoFolderName(videoId) { return sanitizeForFilesystem(videoId); } +// Non-YouTube video identity/path hash: sha256("${extractorKey}:${id}"), +// truncated to 16 hex chars (64 bits) -- short enough to keep the deeper +// NonYT//// path well under Windows' MAX_PATH=260, +// and 64 bits is far past any realistic collision risk for a single user's +// personal archive. Falls back to hashing originalUrl when id is missing, +// since id uniqueness isn't guaranteed by every yt-dlp extractor. +export function nonYoutubeVideoHash(extractorKey, id, originalUrl) { + const input = id ? `${extractorKey}:${id}` : originalUrl; + return crypto.createHash('sha256').update(input).digest('hex').slice(0, 16); +} + // /clips/clips.json -- one JSON array of clip records per video. // Kept minimal: title/extension are derivable from the clip's own filename; // this only holds what scanLibrary/ClipCollectionView need cheaply without // re-invoking ffprobe on every scan. -// Record shape: { id, fileName, title, createdAt, durationSeconds } +// Record shape: { id, fileName, title, createdAt, durationSeconds, clipTimestamps } +// clipTimestamps holds the exact start/end timestamp strings ffmpeg was +// invoked with to produce the clip -- not recomputed from durationSeconds, +// since that's a post-hoc length, not the original in/out points -- so +// ClipCollectionView can hand them back to the original video's player +// (see "Mark clip on original video"). function clipsManifestPath(videoDir) { return path.join(videoDir, CLIPS_DIR_NAME, 'clips.json'); } @@ -373,7 +395,7 @@ export function listClips({ libraryDir, videoDir }) { // Throws on a duplicate fileName rather than silently overwriting or // auto-renaming (product decision); the IPC handler turns this into an // inline dialog error for the renderer. -export function recordClip({ libraryDir, videoDir, fileName, title, durationSeconds }) { +export function recordClip({ libraryDir, videoDir, fileName, title, durationSeconds, clipTimestamps }) { const resolvedVideoDir = resolveInsideLibrary(libraryDir, videoDir); if (!resolvedVideoDir) { throw new Error('Refusing to record a clip outside the configured library folder.'); @@ -382,7 +404,7 @@ export function recordClip({ libraryDir, videoDir, fileName, title, durationSeco if (manifest.some((c) => c.fileName === fileName)) { throw new Error('A clip with this name already exists for this video.'); } - const clip = { id: crypto.randomUUID(), fileName, title, createdAt: Date.now(), durationSeconds }; + const clip = { id: crypto.randomUUID(), fileName, title, createdAt: Date.now(), durationSeconds, clipTimestamps }; writeClipsManifest(resolvedVideoDir, [...manifest, clip]); return clip; } @@ -442,15 +464,18 @@ export function deleteClip({ libraryDir, videoDir, clipId }) { return { success: true }; } -// Shared by writeLibraryEntry (new video) and addLibraryVersion (new version -// of an existing video) so both ever build exactly one metadata shape -- -// two independent inline copies would be free to drift apart over time. -function buildEpochMetadata(videoMetaData, addedEpoch) { - const { id, title, fullTitle, description, thumbnail, originalUrl, duration, durationString, uploadDate, channelId, uploader, resolutions } = videoMetaData; +// Fields every entry shares regardless of platform -- factored out so +// buildYoutubeEpochMetadata/buildGenericEpochMetadata below build on the same +// base instead of two independently drifting copies. uploaderId/timestamp/ +// license/categories/tags/music are schemaVersion 5 additions (see +// SlothArchiver-dossier's non-YouTube platform support plan) captured +// uniformly off reshapeVideoInfo's own shape for every platform, YouTube +// included, even though only generic entries' UI surfaces them today. +function buildCommonEpochMetadata(videoMetaData, addedEpoch) { + const { id, title, fullTitle, description, thumbnail, originalUrl, duration, durationString, uploadDate, uploader, resolutions, uploaderId, timestamp, license, categories, tags, music } = videoMetaData; return { schemaVersion: CURRENT_VIDEO_SCHEMA_VERSION, videoId: id, - channelId: channelId || null, channel: uploader || null, title: title || null, fullTitle: fullTitle || null, @@ -462,8 +487,8 @@ function buildEpochMetadata(videoMetaData, addedEpoch) { uploadDate: uploadDate || null, addedEpoch, // Captured at add-time, not fetched live at download-time -- can go - // stale if YouTube changes available qualities later. Entries written - // before this field existed just won't have it. + // stale if the source changes available qualities later. Entries + // written before this field existed just won't have it. resolutions: resolutions || [], // Adding a video/version and downloading its file are separate // actions -- filled in by recordLibraryDownload() once a download @@ -471,15 +496,57 @@ function buildEpochMetadata(videoMetaData, addedEpoch) { downloadedFilePath: null, downloadedResolution: null, downloadedFormat: null, - // MP3 is a separate, coexisting artifact -- its own slot (audio.mp3, - // alongside video.), independent of the video fields above. - downloadedAudioFilePath: null, lastPlaybackPositionSeconds: null, + uploaderId: uploaderId || null, + timestamp: timestamp || null, + license: license || null, + categories: categories || null, + tags: tags || null, + music: music || null, + }; +} + +// YouTube entries keep channelId -- the one identity field genuinely +// YouTube-specific -- and MP3 as a separate, coexisting artifact (its own +// downloadedAudioFilePath slot, audio.mp3 alongside video.). +function buildYoutubeEpochMetadata(videoMetaData, addedEpoch) { + return { + ...buildCommonEpochMetadata(videoMetaData, addedEpoch), + platform: videoMetaData.platform || 'youtube', + channelId: videoMetaData.channelId || null, + downloadedAudioFilePath: null, + }; +} + +// Generic (non-YouTube) entries have no channelId concept -- "who made this" +// is channel/uploaderId instead -- and no separate audio slot: an audio-only +// download (e.g. SoundCloud's only option) writes into the same +// downloadedFilePath/downloadedResolution/downloadedFormat fields any other +// resolution choice would, so it plays through the one real library player +// instead of a bare native