Repository navigation
librivox.fetch: add a LibriVox recording from archive.org to a books library - #68
Merged
Merged
Conversation
…ry redirect Request and StreamRequest take an optional Allow, asked about the first URL and each redirect hop in CheckRedirect; a refusal fails as blocked_host before the refused host is dialled. Nil keeps today's rule, so podcasts are unchanged. librivox.fetch uses it to stay on archive.org. Part of LibriVox L6-1 (Add from LibriVox).
… jobs importjob.AllBooks now reads a library's paged book listing for audiblejob and the coming librivoxjob, and placement exports Sanitize so a LibriVox fetch names its section files by the same rule as its book folder. No behaviour change. Part of LibriVox L6-1 (Add from LibriVox).
LibrivoxAudiobook reads one project from meta's LibriVox-shaped feed (extended=1&coverart=1: sections, cover, url_iarchive) and LibrivoxWork the work's title, authors and recordings' librivox_id and archive_id. Both are uncached and tell disabled, not found and unavailable apart, which librivox.fetch maps to metadata_off, not_found and a retryable error. Part of LibriVox L6-1 (Add from LibriVox).
…to a books library The job reads the project's manifest from the Bridge's own meta base, fetches each section over https from archive.org only (every redirect checked), stages them in a dot folder beside the destination with Range/If-Range resume, and places Author/Title/NN - Section.mp3, cover.jpg and .audiosilo/librivox.json by one no-replace rename before a rescan. The title's folder taken, it uses Title (LibriVox <id>) once, then fails exists. A cancel or failure removes the staging folder; a stop or lost claim keeps it. Its plan and placement marker live in <data>/librivox so a failed rescan resumes there. Registered on the download lane, so the server sees the librivox feature; the page labels the kind. Part of LibriVox L6-1 (Add from LibriVox).
… section Without a strong ETag a cut attempt restarts the file from zero, so it counted as progress and a flaky link could loop for ever; an attempt now adds only when the partial holds more than before it. The mapping is checked before each section's writes, so one that goes away mid-download parks not_writable with what was fetched instead of failing and deleting it. Part of LibriVox L6-1 (Add from LibriVox).
Fetch and Stream each spelled the redirect's allowlist check inline, beside the shared check of the first URL. Both now call one helper, so the nil rule and the refusal have a single spelling on a security control. Part of LibriVox L6-1 (Add from LibriVox).
LibrivoxWork decoded works/{id} into a second hand-written struct, though the package keeps one mirror of that payload. The payload is now a named type read by fetchWork: workDetail caches its projection and LibrivoxWork projects the same decode uncached. The two reads share their cancel / unavailable / not-found mapping, and the fields no caller read (the work's id, a section's number and playtime) are gone. The fetch test's fake checks the first URL and a redirect with one helper.
Part of LibriVox L6-1 (Add from LibriVox).
…oot helpers The state stored the staging folder beside the parent it is always derived from, and policed the copy on load and before removal; it is computed now. Cleaning up another library's stale fetch went through Abandon with a made-up job; both call discard. plan and place build their two candidate folders without index tests or an empty-string sentinel, plan keeps the author folder's name in its message when the disk can't be read, opening the library's root and the metadata_off failure have one spelling each, transient failures are netguard.Retryable's, and a finished section no longer costs a second state write (its file is the record). The docs name podcast.fetch as sharing the lane and pkg/match as a dependency. Part of LibriVox L6-1 (Add from LibriVox).
A body arrives in reads of one TLS record (16 KB) at most and each became its own write on the .part file: several thousand writes per section, each a round trip on an SMB, NFS or FUSE mount. A 1 MiB buffer sits before the file now, flushed after a cut connection too so the partial still holds everything that arrived for the resume. Part of LibriVox L6-1 (Add from LibriVox).
…al held An attempt counted as progress when it left the partial larger than it found it. Without a strong ETag a cut attempt starts the file again from zero, so cuts at 10 MB, 5 MB, 6 MB, 5 MB, 6 MB each 'added' to the one before and the section retried for ever, holding the download lane. section now keeps the most the partial has held and only an attempt that passes it resets the count. The count also started at one after an attempt that added, so two attempts that added nothing failed the section where the rule says three. Part of LibriVox L6-1 (Add from LibriVox).
…e saved A 200 that starts a file again recorded its new ETag and size in the state before the old .part was truncated. A stop between the two left the old bytes under the new validators: the resume asked Range with the new ETag, got a 206 the Bridge could prove, and appended the new tail to the old head, placing a corrupt section of the right size. The partial is removed first now. A strong ETag is also recorded only beside a stated size: without one no 206 can be proven, so the resume only spent an attempt on a Range it then had to throw away. Part of LibriVox L6-1 (Add from LibriVox).
A 200 with Content-Length 0, or with no stated length and no bytes, skipped the size check: an empty NN - Section.mp3 was renamed into place and skipped on every resume. Zero bytes is a short body now, tried again and then archive_unavailable. Part of LibriVox L6-1 (Add from LibriVox).
…s mount moves The per-section check threw away the mount mapped returned, and download held one handle on the library opened before any park. A library remapped to another folder mid-fetch, or a share remounted while the job was parked, kept the job writing through the old handle: the book was placed where the server can't see it, or a write failed and the cleanup removed everything fetched. recheck now fails not_writable when the mapping's mount is no longer the one the staging folder is in, the library is opened per section, and the cover and the marker get the same check the sections and the rename had. Part of LibriVox L6-1 (Add from LibriVox).
place renamed whatever the staging folder held, and its guard counted every regular file there (the cover, a .part), so it proved nothing about the sections. A run stopped mid-section and resumed after meta's section titles changed placed the old '01 - Old Title.mp3' and a leftover .part beside the new files, and the server indexed the duplicates. sweep now removes what isn't this manifest's sections, the cover or .audiosilo/, and fails the placement when a section is missing. An entry is kept by its name or by being the same file as the one found under that name (a mount may list NFD for NFC), and the sections are looked for again after the removals so an incomplete book is never placed. Part of LibriVox L6-1 (Add from LibriVox).
The rename to the book's folder and the state write that records it are two steps, and a failed write was only logged. A kill or a full data folder between them left a state that said not placed: the next run downloaded the whole book again, found its own folder at the destination and placed a second copy as Title (LibriVox N). A run whose staging folder is gone and whose folder, or its fallback, holds this project's librivox.json is placed now, and goes on from the rescan. Part of LibriVox L6-1 (Add from LibriVox).
… alone The state is keyed by the LibriVox project only, and the server allows one open fetch of a project per library. Abandoning a cancelled job for library A while the state on disk was a later fetch of the same project into library B removed B's staging folder and state. Abandon now returns when the state names another library than the abandoned job's. The package notes also cover the placement adopted from its marker. Part of LibriVox L6-1 (Add from LibriVox).
allowedRaw checked the trimmed URL while the request, and the file's extension, used the string as it stood. A listen_url with a trailing space passed the manifest check and was then requested with %20 appended (a 404), after the staging folder was made; a leading space failed as an invalid URL. A URL with surrounding space is refused where the manifest is read, so the check and the request see one spelling. Part of LibriVox L6-1 (Add from LibriVox).
… per library A meta read that fails for want of service (5xx, a dropped connection) is now tried 4 times inside the run and then fails uncoded with the staging folder and state kept, so the next attempt resumes instead of downloading again; only meta's definite 404 stays terminal as not_found. A section is kept only when it is audio: a response whose Content-Type is a page (text/html, JSON, any non-audio type) is never written, and a whole file whose first bytes are no ID3 tag or MPEG frame sync (or the Ogg, FLAC or MP4 signature) is removed. Both are tried again like a dropped connection, then fail archive_unavailable. netguard's StreamResponse now carries the Content-Type. The state file is keyed by library and project (<library_id>-<librivox_id>.json), so a fetch of one project into a second library no longer discards the first library's staging, and Abandon only touches its own library's fetch. The lost-fold adopt path is unchanged. Part of LibriVox L6-1 (Add from LibriVox).
With state keyed per library and project, Abandon of a library-4 job finds no state at all, so the old test passed without exercising its claim. TestFetchTwoLibrariesIndependent covers Abandon leaving the other library's fetch. Also re-wraps the state comment. Part of LibriVox L6.
mime.ParseMediaType returns the media type beside ErrInvalidMediaParameter for a header such as "audio/mpeg; charset". Treating that error as not audio refused a valid section on every attempt and ended the fetch archive_unavailable, removing its staging. Part of LibriVox L6.
A section whose MP3 frames or ID3 tag follow a few zero bytes failed the audio check on every attempt, ended archive_unavailable and lost its staging. The sniff now skips zero bytes within the first 4 KB before looking for the ID3, MPEG sync, Ogg or FLAC signature; MP4's ftyp is checked first, since its box size leads with zeros. All zeros or a page after padding is still refused. Part of LibriVox L6.
Live verification on project 5495 named a file "01 - 1 - Chapter 01 through Chapter 06.mp3": LibriVox titles some sections with their own number. A title that opens with a number and a separator (" - ", ". ", ": ", ") ") loses that prefix when the number is the section's section_number or its 1-based position; a title that is only that number becomes "Section N", and other numbers ("1984") stay. metaops decodes section_number again for it.
Part of LibriVox L6-1 (Add from LibriVox).
KodeStar
added a commit
to KodeStar/audiosilo-frontend
that referenced
this pull request
Nov 13, 2026
…292) LibriVox L6-2 (v2.7.0 row 14, LIBRIVOX-PLAN.md "L6"), the player side of Add from LibriVox. What: - A books library's Books mode gets a "..." menu with "Find on LibriVox" (admins only). It opens `/find-librivox?connection=&library=`, a Library-tab page beside Find shows. - Search LibriVox works (community metadata, never librivox.org), open a work, see its LibriVox recordings (reader, length, sections, project link). "Add to <library>" queues the server's `librivox.fetch` Bridge job. The row follows the job by id (steps, sections, park and failure reasons), offers Cancel, then "In your library" with Open. Library lists are re-read when a book lands. - Gate `librivoxAccess`: admin + the `bridge` flag + `bridge.features` has `librivox` + `metadata`. Older servers and non-admins never see the menu, and a pasted URL sends no request. Offline state follows the app's offline rule. - Wire mirrored by hand: the three `/admin/librivox` routes, `GET /admin/bridge/jobs/{id}`, error codes `metadata_off`, `in_library`, `not_book_library`. Additive; no change for older servers. - Six locales; CONVENTIONS and STYLEGUIDE entries. Tests: gating, client shapes and error codes, hooks against a fake client (polling, Add, Cancel), the page's states (RNTL), the menu, the route tree. Reviews: /simplify + /code-review high (Fable 5.2) on the whole branch, fixes applied. Verified live (web export under the server's /web CSP, server + paired Bridge from the paired branches, live meta, Chrome at 1440 and 390 px): menu opens the page; Johnny Crow's Garden added and followed to "In your library", Open lands on the book, the book shows in Books mode; Cancel mid-download on a 23-section project leaves no staging folder and offers Add again; a listener, metadata off and a stopped Bridge all hide the menu and send no admin request; no console errors or CSP violations on the page. Needs a device run: the native "..." dropdown, the library Select popover (only shown with two or more books libraries, so not seen live), the LibriVox link on iOS and Android. Pairs with KodeStar/audiosilo-server#378 (routes, job, fold, console) and KodeStar/audiosilo-bridge#68 (librivox.fetch).
KodeStar
added a commit
to KodeStar/audiosilo-docs
that referenced
this pull request
Nov 13, 2026
…ots (#303) Docs for LibriVox L6 (v2.7.0 row 14): Add from LibriVox in the console and the player, run by the Bridge. - New User Guide page `admin/librivox` ("Add free books from LibriVox"): what is needed (admin, community metadata on, a paired Bridge with the library mapped writable), console and player steps, what lands on disk, the book arriving matched, cancel and resume, failure reasons, Clear community matches. - New Developer Docs page `bridge/librivox`: the gate, the three `/admin/librivox` routes, the `librivox.fetch` job (params, progress, result, failure codes), the identity rule (a `book_matches` row from the fold, never a first-recording fallback), archive.org-only fetching, placement, staging and resume, `.audiosilo/librivox.json`. - Updated: features, running a Bridge, Bridge jobs, console tour, books, browsing, FAQ, API reference and index, Bridge pairing/jobs, configuration, attachments, scanner, data model, web UI, notifications, player library/routes/state, the cross-repo contract, the documentation mapping table. Also adds `audible.sync` and `audible.import` where two kind lists had missed them. - Screenshots: `admin/librivox.png` and `web-player/find-librivox.png`, captured from the real UI with live community metadata. Pairs with KodeStar/audiosilo-server#378, KodeStar/audiosilo-bridge#68, KodeStar/audiosilo-frontend#292.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LibriVox L6-1 (v2.7.0 row 14, LIBRIVOX-PLAN.md "L6"), the Bridge side of Add from LibriVox. Bridge Phase 5 is parked, so the job carries its own minimal placement.
What:
internal/librivoxjob, on the download lane (one book at a time, shared with Audible imports and podcast fetches). It reads the project's manifest (/api/v1/librivox/audiobooks?id=&extended=1&coverart=1) and work from the Bridge's own meta base.*.archive.org, checked on the first URL and every redirect (netguard gains a per-request host allowlist), never librivox.org. A section is kept only if its type and first bytes are audio..audiosilo-librivox-<id>folder inside the library. Partial.partfiles resume with Range/If-Range against the ETag. A meta outage is retried and keeps staging for the next run.Author/Title/NN - Section.mp3+cover.jpg+.audiosilo/librivox.json({version:1, librivox_id, work_id, recording_id, archive_id}), moved with one no-replace rename. If the folder is taken it triesTitle (LibriVox <id>)once, then failsexists. LibriVox's own leading section numbers are dropped from file names.placement.Sanitizeexported,importjob.AllBooks,metaopsLibriVox reads, the page's label "Adding from LibriVox".Tests: names (traversal, dot-leading, over-long, empty, Various, padding, LibriVox number prefixes), host allowlist incl. redirects, resume and corrupt-splice cases, audio sniffing (HTML, padding, charset parameters), exists fallback, cleanup on cancel vs stop, per-library state, adopt after a lost fold.
Reviews: /simplify + /code-review high (Fable 5.2) on the whole branch, then a fix round at high (Opus) over the meta-outage, audio-sniff and per-library-state fixes.
Verified live against a scratch server built from KodeStar/audiosilo-server
feat/librivox-l6and the real archive.org: project 192 placed exactly as above,kill -9after section 2 of 8 resumed with the finished sections untouched (same inodes), a mid-section.partresumed from 3 MB with an md5 equal to archive.org's. The Linuxrenameat2path was read, not run (verification ran on macOS).Pairs with KodeStar/audiosilo-server (routes, job kind, fold, console).