Repository navigation
Conversation
…s, regression replay Maintained fork of myxdvz/booktree (upstream inactive since 2026-03). This commit changes no runtime behaviour; it sets the project up for public maintenance: - image ghcr.io/mancolt/booktree built for amd64+arm64 with SBOM/provenance; base pinned to alpine:3.21; rapidfuzz pinned so fuzzy scores cannot drift - CI: ruff, unit tests, bandit, pip-audit, gitleaks, Trivy image scan; Dependabot for pip, actions, docker - SECURITY.md, CONTRIBUTING.md (compatibility promise, never write under the source path, MAM call budget) - docs/FORK.md: output contract kept byte-compatible, defect roadmap mapped to upstream issues - tests/replay: offline replay of historical run logs (search keys, cached Audible/MAM ranking, target paths, OPF, stdout transcripts) + compare tool; frozen baseline summary for the upstream code - .claude/agents: correctness/efficiency and security review agents used before every commit Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JdymXZSW4aRWMFREPAFzkZ
…-arch PR builds, authenticate Trivy pull Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JdymXZSW4aRWMFREPAFzkZ
Trivy flagged Alpine's py3-setuptools 70.3.0 (CVE-2025-47273) and py3-msgpack (GHSA-6v7p-g79w-8964), both pulled in only by 'apk add py3-pip'. The venv seeds its own pip via ensurepip, so the system packages are unnecessary; the venv's pip is removed after installing requirements so the runtime image carries no installer. Also 'apk upgrade' the base and pin build args to their CI defaults. Runtime unchanged: Python 3.12, same dependency versions; corpus replay identical to baseline. Claude-Session: https://claude.ai/code/session_01JdymXZSW4aRWMFREPAFzkZ Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…ss template substitution (#3) Audiobookshelf discards the whole metadata.opf (ASIN included) when a text node contains a bare '&', e.g. publisher 'Little, Brown & Company' (The Tipping Point, 2026-09-06). Upstream inserted metadata into the XML template verbatim via re.sub. - xmlText / xmlAttr / xmlCData helpers: & < > escaped in text nodes, quotes also escaped in the calibre:series attributes, ']]>' split inside CDATA (description, genres, tags), characters XML 1.0 forbids stripped before escaping so they cannot smuggle a terminator or entity - all template tokens substituted in ONE regex pass with a callable replacement: inserted metadata is never rescanned (an author named '__DESCRIPTION__' no longer pulls the CDATA-encoded description, unescaped, into the creator element and with it arbitrary OPF elements such as a second ASIN); a backslash sequence in a description ('\1') is inserted literally instead of raising inside re.sub, which used to abort OPF creation - template path resolved relative to the module instead of the cwd - renderOPF(book) split out as a pure function for tests; createOPF signature and error message unchanged - tests/test_opf.py: &, <, quotes in publisher/title/author/narrator/series, CDATA terminators at every boundary, backslashes, control chars, unicode, None and numeric fields, the token-injection input, and byte-identical output for plain metadata Corpus replay vs baseline: 69 of 266 OPF files changed, exactly the 69 that were not well-formed, all well-formed now; the other 197 byte-identical; stdout, cache keys and target paths unchanged. Claude-Session: https://claude.ai/code/session_01JdymXZSW4aRWMFREPAFzkZ Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…on as evidence (#4) Roadmap item 2. - log mode (fix.csv): a non-empty id3-asin is a pinned ASIN. The Audible product for it is accepted even when the id3 title/author disagree (upstream re-validated against id3 and vetoed the match). A skeleton or missing product falls back to the normal title/author search, never to the same ASIN again. - --hints <json> / Config/hints_file: per release (keyed by booktree name, release folder path or any file path) a pinned asin, up to 10 candidate ASINs fetched by ASIN, an expected duration_min, and search title/authors that replace junk id3 for the search only (logged id3-* columns unchanged). Validated strictly at start-up (ASIN pattern, bounds, finite durations, 16 MiB cap, duplicate keys); an invalid file stops the run before anything is processed. - duration as evidence: among results that pass the title/author gate and matchrate, a runtime within 2 min of the expected duration is preferred over a higher fuzzy score; equal scores prefer the closer runtime; with no runtime information the choice is identical to upstream (first highest score). - Audible skeleton answers ({asin, asset_details, is_vvab}, no title) are skipped instead of compared against empty strings (upstream issue #25). - optional Config/pin_max_runtime_delta_min refuses a pin whose runtime is too far off (default off). - fixes found in review: getAltTitle interpolated author/series names into regexes unescaped ('Brad [' aborted the run, crafted names could hang it); the log reader parsed isMatched/isHardLinked with bool() so 'False' read as True and re-logged rows always claimed a hardlink; getRunTimeLength raised on an empty duration. - tests: hints validation/lookup, pickBest semantics, matcher with an offline Audible double (pins, skeleton fallback, candidates, duration ranking, hint overrides, fixid3), end-to-end buildTreeFromLog with a real hardlink. CONFIG.md documents the hints file. Corpus replay vs baseline: no match gained or lost; 5 best-ASIN changes on 2 books, both runtime corrections (Athena Project 521-min file: 370-min -> 523-min edition, which the live run had matched wrongly; Meditations 309.9-min file: 305-min -> 309-min edition among 14). stdout: only added lines (Duration evidence, Ignoring skeleton result); removed only the empty-string comparisons for skeleton results. Claude-Session: https://claude.ai/code/session_01JdymXZSW4aRWMFREPAFzkZ Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…ags; harness fidelity (#5) Roadmap item 3. Upstream searched Audible with the whole file name as the title when the id3 tags were missing ("Megan Fate Marshman - Relaxed.m4b" -> title "megan fate marshman relaxed" -> nothing), and the MAM result for such a file failed the title gate because the file name was compared as the title. - myx_names.py: parseReleaseName recognises Author - Title, Title - Author, Title by Author, Author-Title, Author - Series NN - Title, Series NN - Title - Author, Title, Series Book N, Title [ASIN|ISBN10], Title [Series NN], Last, First order, leading track numbers, and strips (Unabridged), [M4B], bitrates, recent trailing years and "X narrator" fragments. ASINs are B0xxxxxxxx or ISBN-10 only (never a 10-letter word). isJunkTitle/isJunkAuthors detect empty, AudioTrack-01-style and file-name-repeating tags. On the 202 corpus names with an Audible ground truth: 193 titles and 115 of 120 extracted authors correct. - matching (flags/parse_names, default on; --legacy-names restores upstream): where the tags are junk the parsed title/authors/ASIN drive the search, title and author as separate fields, and a result must agree with the parsed title (a same-author book with another title is rejected). Attempts, each cached and bounded: parsed -> release-name authors (when the tag names someone else, e.g. the narrator) -> swapped reading -> title only (title-verified result accepted on runtime alone) -> upstream's own search unchanged, so a wrong parse can add but never lose a match. Books with usable tags take exactly one attempt and keep upstream's output byte for byte; interactive mode and hint title/authors bypass parsing. The MAM ranking uses the same parsed values; the MAM query string is unchanged. Disc folders (cd1/) are parsed from their release folder. - hardening from review: no adjacent optional-whitespace quantifiers in the junk regexes (a 5000-space id3 tag stalled a run for hours), names capped at 512 chars, malformed AUDIBLE_ASIN tags ignored instead of hitting the per-ASIN URL, non-UTF-8 file names no longer abort the cache-key hash, author/series names escaped in regexes (previous step), duplicate identical attempts skipped. - CI: main publishes :edge; :latest and version tags come from v* tags only. README states pre-release. - replay harness: BOOKTREE_CODE selects the code tree (the working tree used to override the image's code, so an image-vs-image A/B compared identical code); --audible-live / BOOKTREE_REPLAY_LIVE fetches uncached Audible queries into a staged cache copy for fair A/Bs; matched title/authors/runtime recorded; file paths stored source-relative as booktree does (isMultiBookCollection depth), so the frozen baseline is regenerated from upstream code and pass 1 now feeds MAM metadata into the Audible search as production does. Corpus: vs merged main (offline) 2050 transcripts changed, every one a book whose tags needed the release name, 0 good-tag books changed. Live A/B vs upstream code with real Audible: pass 1 +867 and pass 2 +959 book-runs matched, 0 lost; every gain title-verified; the only ASIN choices that differ from both production and upstream are the runtime corrections from the previous step. Claude-Session: https://claude.ai/code/session_01JdymXZSW4aRWMFREPAFzkZ Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
#6) * cache: TTLs for search answers, MAM pacing and budget, --refresh, --json-log Roadmap item 4. Upstream cached every Audible answer forever (a transient or malformed query became a permanent miss; the owner pruned the cache weekly) and never cached empty MAM answers (every unmatched release re-asked MAM on every run). - myx_cache: search-cache freshness by age and emptiness, same files and format as upstream: Audible 30 d positive / 6 h empty (a per-ASIN skeleton without a title is empty), MAM 7 d / 24 h (an answer without a snatched entry is empty); Config/cache/*_hours. Entries dated in the future, of the wrong shape or over 16 MiB are expired; entries younger than the shortest TTL are served without parsing. Errors and HTTP failures are never cached; malformed entries are refetched instead of crashing. - MAM traffic rules (hard requirement of the pipeline): every HTTP request to MAM spaced >= 6 s (Config/mam/min_interval_seconds), at most 60 searches per run (Config/mam/max_queries_per_run), knobs validated and clamped with one warning; the session cookie is tested once per run instead of before every search; the startup cookie check is paced too; POST has a timeout; empty MAM answers are cached under the short TTL. - --refresh RELEASE (repeatable) or hint "refresh": true re-processes one release ignoring its cached answers and its processed marker; hints are applied before the processed check; a --refresh that matches nothing warns. - --json-log [PATH]: JSON-lines run log, one record per processed book (files, id3, parsed name, hint, pin, match with the attempt that produced it, runtime delta, target path, hardlinked, every query with its cache key) plus a run record per paths entry. Additive: CSV and stdout unchanged; writing it can never fail the run; non-UTF-8 names are escaped; log files (CSV too) are opened without following symlinks; cache writes go through a unique temp file and rename. Review: two full reviews; 20 findings acted on (non-fatal JSON log on surrogate names, future-mtime and wrong-shape cache entries, knob validation, cookie test once per run, refresh must be boolean, symlink-safe log writers, run-record accounting, replay harness resets MAM counters per replayed run). Tests: 113 pass. Corpus replay step 4 vs step 3: no differences (TTLs/pacing frozen in the harness by design; --honor-ttl to observe them). Image: 0 HIGH/CRITICAL. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JdymXZSW4aRWMFREPAFzkZ * mam: per-run search cap is a runaway guard (3000), spacing is the safety net 400 searches 6 s apart were fine in practice; a 60-search cap would have left most of a large first run unmatched until later runs. Default raised to 3000, documented as a guard against a looping process only. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JdymXZSW4aRWMFREPAFzkZ --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Release-name parsing applied a bracketed ASIN even when the tags were good. requireTitle is only set when the parsed title replaced a junk tag, so a leftover ASIN in the folder name accepted a different book by the same author and filed it under the wrong title. Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mancolt <Mancolt@users.noreply.github.com> Co-authored-by: Mancolt <38768585+Mancolt@users.noreply.github.com>
…me (#8) Release-name parsing from #5 set requireTitle only when the parsed title replaced a junk tag. A usable id3 title with a junk artist still took authors from the folder and then used the author-only gate, so a leftover or wrong Author - Title folder (James Patterson - The Guest, id3 title The Guest) filed a different book by that author (Along Came a Spider, token_sort 71). requireTitle is now set when title or authors were parsed. MAM ranking uses the same gate and compares against the id3/parsed title, not the file basename. Validation: two Audible tests and two MAM ranking tests; 95 unit tests OK. Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mancolt <Mancolt@users.noreply.github.com>
* Add Cloud Agent dev environment config Configure the Cloud Agent development environment for booktree: install ffmpeg (used by ffprobe for id3 metadata) and the Python runtime dependencies plus the CI dev tools (ruff, bandit, pip-audit) so lint, tests and security scans run exactly as in CI. booktree is a CLI, so no start/terminals services are required. Co-authored-by: Mancolt <Mancolt@users.noreply.github.com> * Install python3.12-venv for pip-audit pip-audit builds an isolated venv to resolve requirements.txt, which needs ensurepip from python3.12-venv. It is absent from the base image, so add it to the install step alongside ffmpeg. Co-authored-by: Mancolt <Mancolt@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mancolt <Mancolt@users.noreply.github.com>
When Config/session is empty or omitted, read the session cookie from the MAM_SESSION environment variable. This lets the cookie come from a secret store or shell env instead of a committed config (which .gitignore and CONTRIBUTING forbid). Resolved once in myx_args.Config so every call site (searchMAM, checkMAMCookie) picks it up; an explicit Config/session still wins. Note: MAM sessions are IP/ASN-locked, so this is a local-run convenience only -- it does not help from a Cloud Agent VM, whose egress originates from a datacenter ASN. Adds tests/test_config_session.py and documents the fallback in CONFIG.md. Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mancolt <Mancolt@users.noreply.github.com>
…des, title_patterns fix (#11) MAM session (roadmap 5 and 7): - Config/session may be blank: MAM_SESSION, then the first line of MAM_SESSION_FILE (docker/compose secret) fill it. - mousehole integration reworked from upstream PR #24: Config/mousehole_state_file or MOUSEHOLE_STATE_FILE; reads the v2 schema (`cookie`) and the legacy `currentCookie`; the file is read on every run, never written back. - New myx_session.py: candidates in order mousehole -> <log_path>/cookies.json -> config/env; the first one MAM accepts is used for the whole run and the check is done once. cookies.json (0600, atomic write) replaces cookies.pkl, which was unpickled from a shared directory on every run; an existing pickle is removed, never loaded. - The cookie lives in the session jar under MAM's own domain (.myanonamouse.net) so a Set-Cookie rotation replaces it and is what gets persisted. Session files are read through the descriptor (a planted FIFO or huge file cannot hang the run, RecursionError from deep JSON is caught); values that are not cookie-octets are refused before any request (no header-injection, no secret echoed by http.client). MAM unreachable at start-up keeps the store. - The store follows the same default log path as the CSV log (./logs when Config/log_path is empty). Exit codes: 0 processed, 2 configuration/input problem (config file, Config/paths shape and existence, hints file unreadable or invalid, log-mode input, MAM session), 1 unhandled error (traceback + `run` record with exit_code and error under --json-log), 130 interrupted. Configuration problems print a message, not a traceback. title_patterns: the template's "\bpart\b" was a JSON backspace, not a word boundary, so noise words were never stripped from the alt title; the template is fixed and patterns holding \x08 are repaired at load with a one-time note. Replay vs main: 13 book-runs lose "of"/"part" from the alt search key; live A/B gained 2 matches, lost none. CI: bandit gates on -ll; ruff S301 exemption removed. Docs: CONFIG.md (MAM session sources, exit codes, mousehole caveat), README, CHANGELOG, FORK.md. 163 unit tests; offline replay identical to the previous step-5 tree apart from the reworded cookies.pkl note.
…onfig/dedupe_roots (#12) Everything a hook or timer otherwise needs a wrapper script for, as config (all off until set): - --pin RELEASE=ASIN (repeatable; Config/pins): the release is re-processed now with that Audible product as authoritative (a hint {"asin", "refresh": true}). Malformed pin -> exit 2; unused pin reported at the end; a pinned/refreshed release is scanned even when older than last_scan and filed even when dedupe finds a copy (the copy is the wrong match being corrected). - Config/notify: run summary to an ntfy topic (token via NTFY_TOKEN; on = always|unmatched|failure, default unmatched) and a heartbeat URL fetched after a clean run. Runs on every exit path once the config is loaded, including unhandled errors; can never change the exit code. URLs are never printed; release names and error text in the message are bounded and stripped of control characters; no redirects are followed. - Config/abs: POST /api/libraries/<id>/scan (token via ABS_API_TOKEN / ABS_API_TOKEN_FILE) after a run that created hardlinks; library_id restricted to [A-Za-z0-9_-]. - Config/dedupe_roots: a matched release whose files (all of them, regular files only) are already hardlinked under a listed directory is reported (`already_filed` in the JSON log), logged as matched with its target path, marked processed and not hardlinked again. Nothing is deleted. A root containing or inside a source_path is refused. Book.selectMetadataBook() factored out of createHardLinks so a dedupe-skipped book still gets its CSV `paths`. CSV columns, stdout compat lines and matching untouched: offline replay identical to main. 183 tests.
…le (#13) --pin RELEASE=ASIN --remember (Config/remember_pins) appends each pin whose Audible product was accepted for a release to the configured hints file as {"asin": ...}, without refresh, so the correction persists across cache wipes and --no-cache. The file is rewritten atomically through its real path, mode and owner are kept best-effort, an unparsable or non-regular file is never overwritten, a dry run only announces the write, and a failed write cannot change the exit code. --remember without a hints file exits 2. A pin Audible could not resolve is reported and not remembered. CONFIG.md and README gain a "Correcting a match" section explaining what booktree does and does not re-apply, when to use Audiobookshelf's Match versus a pin, and that the old folder must be removed by hand. Co-authored-by: Cursor <cursoragent@cursor.com>
* grouping: treat cd1/ discs as one release, not one book named cd1 Hybrid mode keyed books on the immediate parent folder, so two downloads that both used cd1/ were merged and hardlinked to the first match, and a single release's discs were matched separately with one-disc runtime. Group by the release folder (walking past cd/disc/part and cd1/MP3/), include every file of a last_scan-hot release, and re-process only when an older sibling disc was pulled in. Co-authored-by: Mancolt <Mancolt@users.noreply.github.com> * grouping: clarify last_scan comment after sibling-only cache bypass Co-authored-by: Mancolt <Mancolt@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mancolt <Mancolt@users.noreply.github.com>
* dedupe: do not treat split library folders as already filed alreadyFiled returned the first inode's folder whenever every file of a release appeared somewhere under dedupe_roots. Discs left in two different matches (a prior multibook run, or a file moved in the library) were skipped, then cacheMe hid the rest of the files on later runs. Require every file to live in the same folder so hardlinkUnlessFiled can complete the release. Co-authored-by: Mancolt <Mancolt@users.noreply.github.com> * dedupe: disc subfolders of one book (Title/cd1, Title/cd2) still count as filed The same-folder rule would have reported every multi-disc book booktree filed itself (Config/target_path/disc_folder) as not filed, switching dedupe_roots off for exactly the releases that #15 now groups. Collapse a disc subfolder to its parent before comparing; a split filing across two titles still fails the check. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mancolt <Mancolt@users.noreply.github.com> Co-authored-by: Mancolt <mancolt86@gmail.com>
…N/ (#16) Since #15 groups cd/disc/disk/part N folders into one book, isMultiCD must recognise the same vocabulary, or Disk 1/01.mp3 and Disk 2/01.mp3 are filed into one flat folder and the second is silently skipped. cd/disc keep their unanchored match; part requires a word boundary so a title like "Counterpart 2" is not taken for a disc. Replay corpus: no differences. Co-authored-by: Cursor <cursoragent@cursor.com>
…r files as "Series # - Title" (#17) New template, default {author}/{series}/{series} - {title}, used when the book is in a series but its part is empty after sanitisation (upstream #27). Configurable like in_series/no_series; setting it to the in_series value keeps the old names. Corpus replay: 12 book-runs (10 distinct titles) change from "Series # - Title" to "Series - Title", nothing else. Co-authored-by: Cursor <cursoragent@cursor.com>
* log mode: series parts survive the round trip through the run log (upstream #27) The log writes seriesparts as "Name part" and the reader split on "#", so a fix.csv row came back with the series named "Pike Logan 17.5" and no part. The reader now pairs seriesparts with the series column, cleansing both the same way, and reads a pre-fix "17 5" back as 17.5. The writer keeps the part verbatim (17.5, not "17 5"). Corpus replay: four novellas' match-rate strings change by the dot, no match changes. Co-authored-by: Cursor <cursoragent@cursor.com> * FORK.md: item 6 closed Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
CHANGELOG: version heading with upgrade notes from myxdvz/booktree:latest; README: status line describes the published tags instead of pre-release. Co-authored-by: Cursor <cursoragent@cursor.com>
#15 walked past cd1/MP3/ but treated Title/MP3/ as the release, so two downloads that both used a codec folder were keyed as "MP3" and hardlinked to the first match. Walk past FORMAT_FOLDER always. Filing also uses the disc parent of cd1/MP3/ so the second disc is not skipped. Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mancolt <Mancolt@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Author
|
accidentally applied to original booktree, not my fork |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.