librivox.fetch: add a LibriVox recording from archive.org to a books library

CLAUDE.md

@@ -157,6 +157,11 @@ internal/
               -> rescan -> ASIN enrich
   audiblesync/ Phase 4c: audible.sync (lastpositions -> furthest-only events) + the push loop (feed -> SetLastPosition)
   importjob/  BookMatcher (link.LibraryBook) + Plan: match and place against a library
+  -- LibriVox L6 --
+  librivoxjob/ librivox.fetch (download lane): the manifest from meta's LibriVox-shaped
+              feed, the sections from archive.org only (https, every redirect checked)
+              into a staging folder, Author/Title/NN - Section.mp3 + cover.jpg +
+              .audiosilo/librivox.json by one no-replace rename, rescan
   transfer/   Local.Place: SafeJoin + os.Root, free-space check, no-replace install
   placement/ source/ progress/  (the manager's; source trimmed to Record in 4b)
 web/          the page: Vite + React 19 + TS + Tailwind v4; pure logic in src/lib
@@ -166,7 +171,7 @@ Dockerfile    slim | cuda | agent | full; compose.example.yaml: server, bridge,
 
 **Dependency direction:** `app -> {api, jobs, link, mapping, verify, generate,
 agentbox, workers, accounts, setup, correct, store, secrets, config, auth, logring,
-web, netguard, podcast, podcastjob, audibleacct, audiblejob, audiblesync}` plus the engine it wires; `accounts -> {agentproto, ghauth, secrets, store}`; `correct ->
+web, netguard, podcast, podcastjob, audibleacct, audiblejob, audiblesync, librivoxjob}` plus the engine it wires; `accounts -> {agentproto, ghauth, secrets, store}`; `correct ->
 {contrib, jobs, metaops}`; `setup -> link`;
 `api` calls the injected Backend and the injected `auth.Manager` (login, sessions,
 the password), with types from jobs/link/mapping/store/logring; `verify -> {jobs,
@@ -177,7 +182,8 @@ through its own Engine/Server interfaces; `podcastjob -> {jobs, link, mapping, n
 imports `audible`); `importjob -> {link, placement, source,
 audiosilo-server/pkg/match}`; `audibleacct -> {audible, link, secrets, source, store}`;
 `audiblejob -> {audible, audibleacct, fsutil, importjob, jobs, link, mapping, placement,
-source, store, transfer}` (app wires both); `audiblesync -> {audible, audibleacct, jobs, link, store}`; `transfer -> {fsutil, mapping, progress}`;
+source, store, transfer}` (app wires both); `librivoxjob -> {fsutil, importjob, jobs, link,
+mapping, metaops, netguard, placement, transfer, audiosilo-server/pkg/match}`; `audiblesync -> {audible, audibleacct, jobs, link, store}`; `transfer -> {fsutil, mapping, progress}`;
 `jobs -> {link, store}`; `mapping -> {fsutil, link}`;
 `link`, `agentproto` and `workerproto` are leaves; `workers -> {asr, fsutil,
 scheduler, state, store, transcript, workerproto}`; `worker -> {asr, fsutil, workerproto}`
@@ -205,7 +211,10 @@ in PACKAGES.md.
   book writer is `generate` (its three jobs): `characters.json`, `recaps.json`,
   `description.json` (Phase 1c), `chapters.json` (Phase 1d), `readalong.json` (Phase
   3c-1) and `bridge.json` in the book's `.audiosilo/` (a single-file book's prefixed
-  `<stem>.`). Podcast writers
+  `<stem>.`). `librivox.fetch` (LibriVox L6) writes whole new book folders: its own
+  dot staging folder beside the destination (removed outright on a cancel or a
+  failure: it holds only the job's own downloads), then one no-replace rename,
+  never into an existing book. Podcast writers
   require a podcast-kind library and its checked marker mapping (a job claimed
   before the check waits for it); they write only the root marker, show folders,
   feed.json and sniffed JPEG/PNG artwork. Book writers refuse podcast libraries.
@@ -440,6 +449,14 @@ key set) with Range, a cut and an expiry. No migration.
 server's still places back to Audible; a refused account neither syncs nor pushes,
 a sync error never refuses one; the page's sync switch went (the console owns it).
 
+**LibriVox L6-1 (Add from LibriVox, Bridge side):** `librivox.fetch`
+(`librivoxjob`, LIBRIVOX-PLAN.md L6): the manifest from the Bridge's own meta base,
+the sections from archive.org only (`AllowedURL` on every URL and redirect through
+netguard's new per-request `Allow`), staged in the library and placed by one
+no-replace rename with `cover.jpg` and `.audiosilo/librivox.json`, then rescanned;
+resumable per section, its plan and placement marker in `<data>/librivox/`. The
+server records the book's LibriVox identity from the job's result (its fold).
+
 See BRIDGE-PLAN.md's Status table for what is next.
 
 Operative caveats carried from sidecars (PACKAGES.md has the detail):

PACKAGES.md

@@ -1734,7 +1734,12 @@ internal/
             landing in between is never cleared nor parked over.
   metaops/  meta.audiosilo.app client (coverage/lookup, capped 1h TTL caches,
             graceful degrade) + the PathAllowed/AllowedPath containment checks
-            (Phase 1a passes the mapped mounts as the roots). glossary.go adds
+            (Phase 1a passes the mapped mounts as the roots). librivox.go (LibriVox
+            L6): LibrivoxAudiobook (the LibriVox-shaped feed, one project with
+            extended=1&coverart=1; LibriVox's string ids, nullable coverart_jpg,
+            ArchiveID from url_iarchive) and LibrivoxWork (works/{id}: title,
+            author names, each recording's librivox_id and archive_id), uncached,
+            with ErrDisabled / ErrNotFound / ErrUnavailable apart. glossary.go adds
             SeriesGlossary (work -> its series -> the OTHER volumes' contributed
             character names/aliases, <= 12 siblings, own TTL caches), the canonical
             spelling evidence the spelling stage checks a transcript against; it
@@ -2036,6 +2041,12 @@ internal/
             failures, respecting Retry-After. Bad gzip data (empty, header,
             checksum, stream) fails at once as not_a_feed; a read the connection
             broke is retried. Errors never contain remote text or URLs.
+            LibriVox L6: Request and StreamRequest take an optional Allow, asked
+            about the first URL and every redirect's (CheckRedirect); a refusal is
+            blocked_host before the refused address is dialled. Nil keeps the
+            old rule (the podcasts pass none).
+            StreamResponse carries the Content-Type as sent (librivox.fetch refuses
+            a page where a file was asked for).
   podcast/  pure feed.json v1 types, rooted regular-file IO with capped atomic
             writes, URL normalization/hash/privacy/redaction, NFC show names,
             stable episode paths, identity merge and adaptive polling. Spike 5
@@ -2589,12 +2600,115 @@ internal/
             the library holds a record is BookMatcher's). Run, the existence check
             and ApplyDestinations went in Phase 4b: audiblejob's import runs the
             steps. The manager's serverapi client went: the Bridge holds
-            the paged listing. Deps: link, placement, source,
+            the paged listing. AllBooks (books.go, LibriVox L6) reads that listing
+            whole for audiblejob and librivoxjob. Deps: link, placement, source,
             audiosilo-server/pkg/match.
   placement/ where a new book goes (the manager's): Auto (series siblings'
             folder and shortcode convention, else author/title) and templates.
             Phase 4b: a generated segment loses its leading dots too ("...And
             Ladies of the Club", ".hack"): a hidden name is never indexed.
+            LibriVox L6 exports Sanitize (librivoxjob names its section files
+            with it).
+  librivoxjob/ LibriVox L6 (LIBRIVOX-PLAN.md), librivox.fetch on the download lane
+            (capacity 1, shared with audible.import and an ordinary podcast.fetch:
+            one book at a time). Params
+            {librivox_id, work_id, recording_id, library_id} (server-built,
+            strict; unknown fields ignored). Manifest from the Bridge's OWN meta
+            base (metaops.LibrivoxAudiobook: /api/v1/librivox/audiobooks?id=N&
+            extended=1&coverart=1, LibriVox's shape; decision 5, the server's
+            mirror is never asked) and the work (metaops.LibrivoxWork, works/{id};
+            a 404 there is no failure, the project names the folder then). meta
+            unavailable (5xx, a dropped connection) is never terminal: each read
+            is tried 4 times (5, 10, 20 s apart), then the run fails uncoded
+            (keepStaging) with its staging folder and state kept for the next
+            attempt; only meta's definite 404 ends the fetch as not_found. Every
+            section URL must be https on archive.org or *.archive.org
+            (AllowedURL, checked before anything is fetched or written, and on
+            every redirect through netguard's per-request Allow; never
+            librivox.org; a padded URL is refused as it stands, so the check
+            and the request see one spelling), fetched by the app's own netguard client (public
+            addresses only, not the podcasts' allow_networks). Placement:
+            placement.Suggest standalone (an existing author folder from
+            importjob.AuthorFolders reused verbatim, else the sanitized author),
+            NOT importjob.Planner.Plan: its match.CleanTitle drops "(version 2)"
+            and "Vol. 001", which tell two projects of one work apart. Title =
+            the work's, or the project's own when the work has more than one
+            LibriVox recording; author = the first credit, Various over three;
+            no author places at the library root; nothing left of a title is
+            "LibriVox <id>". Names: control characters and space runs folded,
+            Sanitize, cut to 180 bytes on a rune boundary; files "NN - <section
+            title>.<ext>" (NN the 1-based position, at least two digits, the
+            count's width; never section_number; a title opening with the
+            section's own number and a separator ("1 - ", "1. ", "1: ", "1) ",
+            matching its section_number or position) loses it, so 5495's "1 -
+            Chapter 01 ..." is "01 - Chapter 01 ...", while "1984" stays; an
+            untitled section, or one that was only its number, is "Section N"; the URL's audio extension, else .mp3). The title's
+            folder taken, once "<Title> (LibriVox <id>)", else exists; a folder
+            already holding this project's librivox.json is the book (rescan
+            only); an author folder that is a listed book or holds audio is
+            exists. Staging <parent>/.audiosilo-librivox-<id>/ (a dot folder the
+            scan skips), every write through the mount's os.Root (0755/0644):
+            each section to <name>.part, its strong ETag and size recorded in
+            the state before the body (the ETag only beside a stated size; a
+            body from its start removes the old partial BEFORE the new
+            validators are saved, so a stop between the two never leaves old
+            bytes a later 206 would be proven to continue), a resume asks
+            Range/If-Range (netguard.ResumeAction proves the 206; anything
+            unproven starts the file again), renamed to its name only at the
+            stated size and never empty, and only audio (a Content-Type other
+            than none, audio/*, octet-stream or application/ogg is never written;
+            a whole file whose first bytes (past zero padding in its first 4 KB)
+            are no ID3 tag, MPEG frame sync, OggS, fLaC or MP4 ftyp is removed;
+            both tried again like a dropped
+            connection, then archive_unavailable; a finished file is skipped on
+            resume);
+            three attempts in a row that add nothing fail archive_unavailable
+            (adding = past the MOST the partial has held, section's high-water
+            mark, so restarts from zero that only pass the last cut don't
+            count), as does a 4xx other than 408, 416 and 429 (tried again);
+            the library's free space 1.1x each file's rest, else disk_full.
+            The mapping is checked again before each section, before the cover
+            and marker, and before the rename (recheck): not writable parks; a
+            mapping moved to another mount fails not_writable (the staging
+            folder isn't there), and the library is opened per section, not
+            held across a park. cover.jpg best effort (only a
+            sniffed JPEG from archive.org), then .audiosilo/librivox.json
+            {version 1, librivox_id, work_id, recording_id, archive_id} (decision
+            3), the mapping checked again (a resumed or late job parks
+            not_writable), the staging folder swept to this manifest's files
+            (sweep: a partial or a section an earlier run staged under a title
+            the manifest changed since is removed, a missing section fails; an
+            entry is the manifest's by its name or by os.SameFile with the file
+            found under that name, since a mount may list NFD for NFC, and the
+            sections are looked for again after the removals),
+            ONE fsutil.RenameNoReplace staging -> book folder (a
+            folder that raced in takes the fallback once, never replaced), then
+            the rescan (3 tries). State <data>/librivox/<library_id>-<id>.json,
+            one per library and project, so fetches of one project into two
+            libraries never touch each other's staging (state.go: the
+            library, parent and whether the fetch made it, destination (staging is
+            <parent>/.audiosilo-librivox-<id>, never stored),
+            the partials' validators, then placed + count + bytes): a resumed run
+            follows it rather than planning again, Abandon finds the staging
+            folder by it, and once placed it is the placement marker (a failed
+            rescan keeps the book and the state; the retry goes on from the
+            rescan). A placement the state never recorded (a kill between the
+            rename and the state's write) is adopted by the marker: staging
+            gone and the folder or its fallback holding this project is placed
+            (adopt), never fetched again. A stop or a lost claim keeps staging
+            and state; a cancel, a
+            failure or Abandon (not on a lost claim; only the abandoned job's own
+            library's state) removes
+            the staging folder, the author folder it made when empty, and the
+            state. A lost claim
+            never handed out again leaves its staging folder (hidden, unindexed).
+            Steps manifest, download (done/total = sections), cover, place,
+            rescan. Result {library_id, path (the rescan's), librivox_id, files,
+            bytes} (files and bytes: the sections and the cover); a failure is
+            Failure {code, message} as result and error, codes not_writable,
+            exists, disk_full, not_found, archive_unavailable, metadata_off,
+            never a URL. Deps: fsutil, importjob, jobs, link, mapping, metaops,
+            netguard, placement, transfer, audiosilo-server/pkg/match.
   source/   Record (normalized metadata of one importable book: the ASIN,
             UnavailableReason; Phase 4a: Authors (every credit), SeriesPosition as
             the source spells it, ReleaseDate YYYY-MM-DD or "") (source.go only;
@@ -3553,5 +3667,6 @@ and the shared clip-floor predicate `qa.ClipStartInRange`/`qa.ClipStartFloorSec`
 `state` is pure. Phase 4a: `audibleacct -> {audible, link, secrets, source, store}`
 and `audiblejob -> {audible, audibleacct, fsutil, importjob, jobs, link, mapping, placement,
 source, store, transfer}` (Phase 4b's import; app wires both;
-neither imports app or api). Handlers marshal DTOs and call into the injected packages; they
+neither imports app or api); LibriVox L6: `librivoxjob -> {fsutil, importjob, jobs, link,
+mapping, metaops, netguard, placement, transfer, audiosilo-server/pkg/match}` (app wires it). Handlers marshal DTOs and call into the injected packages; they
 hold no logic (state transitions live in `state`, dispatch in `scheduler`).

internal/app/app.go

@@ -75,6 +75,9 @@ type Bridge struct {
 	runner      *jobs.Runner
 	auth        *auth.Manager
 	podcastHTTP *netguard.Client
+	// librivoxHTTP is librivox.fetch's guarded client: public addresses only
+	// (never the podcasts' allow_networks), archive.org hosts only per request.
+	librivoxHTTP *netguard.Client
 	// box is the agent container runner (decision 15), gen the sidecars.generate
 	// job, engine the sidecars engine once it has started.
 	box    *agentbox.Runner
@@ -216,7 +219,8 @@ func New(ctx context.Context, opts Options) (*Bridge, string, error) {
 		}
 		return nil
 	}})
-	corrector := correct.New(correct.Options{Meta: metaops.NewClient(cfg.Metadata.BaseURL), Tokens: b.tokens,
+	meta := metaops.NewClient(cfg.Metadata.BaseURL)
+	corrector := correct.New(correct.Options{Meta: meta, Tokens: b.tokens,
 		BaseURL: cfg.Contribution.APIBaseURL, CoreRepo: cfg.Contribution.CoreRepo, KV: db, ServerID: serverID, Log: opts.Logger})
 	var audiblePoll *audiblejob.Handler
 	var audibleImport *audiblejob.Importer
@@ -225,7 +229,7 @@ func New(ctx context.Context, opts Options) (*Bridge, string, error) {
 	audibleSync, b.audiblePush = b.newAudibleSync(db, opts.Logger, serverID)
 	b.runner = jobs.New(db, b.server, b.tokenRefused, jobs.Options{Log: opts.Logger, ServerID: serverID},
 		verifier, b.gen, b.gen.DescribeJob(), b.gen.ChaptersJob(), b.gen.ReadalongJob(), b.gen.ContributeJob(), corrector, podcasts, podcasts.PollJob(), podcasts.FetchJob(), podcasts.RetainJob(),
-		audiblePoll, audibleImport, audibleSync)
+		audiblePoll, audibleImport, audibleSync, b.newLibrivox(meta, opts.Logger))
 	conn.onChange = b.linkChanged
 	return b, oneTime, nil
 }
@@ -298,6 +302,7 @@ var listen = func(ctx context.Context, addr string) (net.Listener, error) {
 // or the page's server fails (its error is returned once everything stopped).
 func (b *Bridge) Run(ctx context.Context) error {
 	defer b.podcastHTTP.Close()
+	defer b.librivoxHTTP.Close()
 	defer func() { _ = b.db.Close() }()
 	// The heartbeat and the runner stop with ctx: a failing page server must
 	// stop them too, or Run would wait for them for ever.

internal/app/app_test.go

@@ -144,7 +144,7 @@ func TestPairBeatMapAndRevoke(t *testing.T) {
 		return ok && len(st.Libraries) == 1 && st.Libraries[0].Mapped && st.Libraries[0].Writable
 	})
 	st, _ := fs.lastBeat()
-	if st.Version != "0.1.0" || strings.Join(st.Kinds, ",") != "audible.import,audible.poll,audible.sync,bridge.verify,chapters.detect,meta.correct,podcast.fetch,podcast.poll,podcast.retain,podcast.subscribe,readalong.align,sidecars.contribute,sidecars.describe,sidecars.generate" || st.UIURL != "http://bridge.lan:8091" {
+	if st.Version != "0.1.0" || strings.Join(st.Kinds, ",") != "audible.import,audible.poll,audible.sync,bridge.verify,chapters.detect,librivox.fetch,meta.correct,podcast.fetch,podcast.poll,podcast.retain,podcast.subscribe,readalong.align,sidecars.contribute,sidecars.describe,sidecars.generate" || st.UIURL != "http://bridge.lan:8091" {
 		t.Fatalf("heartbeat = %+v", st)
 	}
 	// Once the engine runs, every heartbeat carries the estimates.
@@ -283,7 +283,7 @@ func TestEngineViewAndJobRoutes(t *testing.T) {
 	if _, err := b.Job(context.Background(), 5); !errors.Is(err, api.ErrNotFound) {
 		t.Fatalf("an unknown job = %v", err)
 	}
-	if k := strings.Join(b.runner.Kinds(), ","); k != "audible.import,audible.poll,audible.sync,bridge.verify,chapters.detect,meta.correct,podcast.fetch,podcast.poll,podcast.retain,podcast.subscribe,readalong.align,sidecars.contribute,sidecars.describe,sidecars.generate" {
+	if k := strings.Join(b.runner.Kinds(), ","); k != "audible.import,audible.poll,audible.sync,bridge.verify,chapters.detect,librivox.fetch,meta.correct,podcast.fetch,podcast.poll,podcast.retain,podcast.subscribe,readalong.align,sidecars.contribute,sidecars.describe,sidecars.generate" {
 		t.Fatalf("kinds = %s", k)
 	}
 }