From e535f3040de4abe07009af5b14019d226b4e1a87 Mon Sep 17 00:00:00 2001 From: Mancolt Date: Mon, 14 Sep 2026 19:58:38 -0400 Subject: [PATCH 1/3] Fork booktree as ghcr.io/mancolt/booktree: CI, policies, review agents, regression replay Maintained fork of myxdvz/booktree (upstream inactive since 2026-03). This commit changes no runtime behaviour; it sets the project up for public maintenance: - image ghcr.io/mancolt/booktree built for amd64+arm64 with SBOM/provenance; base pinned to alpine:3.21; rapidfuzz pinned so fuzzy scores cannot drift - CI: ruff, unit tests, bandit, pip-audit, gitleaks, Trivy image scan; Dependabot for pip, actions, docker - SECURITY.md, CONTRIBUTING.md (compatibility promise, never write under the source path, MAM call budget) - docs/FORK.md: output contract kept byte-compatible, defect roadmap mapped to upstream issues - tests/replay: offline replay of historical run logs (search keys, cached Audible/MAM ranking, target paths, OPF, stdout transcripts) + compare tool; frozen baseline summary for the upstream code - .claude/agents: correctness/efficiency and security review agents used before every commit Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01JdymXZSW4aRWMFREPAFzkZ --- .claude/agents/review-correctness.md | 48 +++++ .claude/agents/review-security.md | 55 +++++ .github/dependabot.yml | 11 + .github/workflows/ci.yml | 91 ++++++++ .github/workflows/docker-image.yml | 34 --- .gitignore | 12 ++ CONTRIBUTING.md | 20 ++ Dockerfile | 2 +- README.md | 8 +- SECURITY.md | 12 ++ docs/FORK.md | 47 ++++ requirements.txt | 4 +- ruff.toml | 12 ++ tests/__init__.py | 0 tests/replay/baseline-summary.json | 37 ++++ tests/replay/compare.py | 63 ++++++ tests/replay/replay_corpus.py | 310 +++++++++++++++++++++++++++ tests/replay/run_replay.sh | 21 ++ tests/test_output_contract.py | 30 +++ 19 files changed, 780 insertions(+), 37 deletions(-) create mode 100644 .claude/agents/review-correctness.md create mode 100644 .claude/agents/review-security.md create mode 100644 .github/dependabot.yml create mode 100644 .github/workflows/ci.yml delete mode 100644 .github/workflows/docker-image.yml create mode 100644 .gitignore create mode 100644 CONTRIBUTING.md create mode 100644 SECURITY.md create mode 100644 docs/FORK.md create mode 100644 ruff.toml create mode 100644 tests/__init__.py create mode 100644 tests/replay/baseline-summary.json create mode 100755 tests/replay/compare.py create mode 100644 tests/replay/replay_corpus.py create mode 100755 tests/replay/run_replay.sh create mode 100644 tests/test_output_contract.py diff --git a/.claude/agents/review-correctness.md b/.claude/agents/review-correctness.md new file mode 100644 index 0000000..114eb54 --- /dev/null +++ b/.claude/agents/review-correctness.md @@ -0,0 +1,48 @@ +--- +name: review-correctness +description: Correctness, accuracy and efficiency reviewer for booktree changes. Use after every code step, before committing, on the diff of the working tree or a commit range. +tools: Read, Grep, Glob, Bash +--- + +You review changes to booktree, a Python tool that matches audiobook files against Audible and MyAnonamouse (MAM) +metadata and hardlinks them into a media library. You are the correctness/accuracy/efficiency reviewer; a separate +agent covers security. Report findings only; do not edit files. + +## What you check, in priority order + +1. **Behavioural correctness of the change.** Trace every changed code path with a concrete input. For matching + changes, reason about the actual data shapes: id3 tags can be empty or junk (`AudioTrack 01`, `unknown artist`), + Audible per-ASIN lookups can return a skeleton `{asin, asset_details, is_vvab}` with no title, MAM `author_info` + / `series_info` are JSON strings, series parts can be decimals (`14.5`) or empty. +2. **Output contract (byte-compatibility).** The run-log CSV must keep the 46 columns from + `myx_utilities.getLogHeaders()` in order, `isMatched`/`isHardLinked` written as `True`/`False`, file name + `booktree_log_YYYYMMDDHHMMSS.csv`. The stdout phrases `Processing: …`, `Searching Audible for` + indented + `asin:`/`title:`/`authors:`/`narrators:`/`keywords:`, `Searching MAM for` + `TitleFilename:`, + `Checking cache: /`, `Caching in File:`, `Found N Audible match(es)`, + `Creating Hardlinks for N matched books`, `Hardlinking files for ` + `from`/`to` must be unchanged unless the + change is explicitly about them. Flag any print() edit that touches these. +3. **Hard operational rules.** Nothing may write, move, rename or re-tag anything under the source path + (`source_path`, seeded torrents). Only hardlink/copy into `media_path`. `fixid3` must default to off. Flag any + `os.rename`, `shutil.move`, tag writing, or file creation whose target could resolve under the source path. +4. **External-call budget.** MAM calls must stay bounded per run and ≥ 6 s apart; Audible calls should not multiply + per book without a cap. Cache keys must remain deterministic (sha256 of the query) so consumers can map them. +5. **Efficiency.** Repeated ffprobe/network/disk work inside loops, O(n²) over the book list, re-reading the same + cache file, regex compiled per row, unnecessary copies of large JSON. Only flag what matters at ~2k files per run. +6. **Tests.** Every fix needs a unit test with the failing input from the bug report (e.g. `&` in publisher, + `Author - Title.m4b`, skeleton ASIN response). Check the test actually exercises the changed code and would have + failed before the change. Check the replay baseline comparison was run when matching/output code changed. +7. **Python correctness.** Mutable default args, mutated dataclass fields shared across instances (upstream `Book` + has `metadata={}` and `matchRate=0` as class attributes), exception swallowing that hides failures, `bool("False")` + truthiness bugs (upstream has one in `buildTreeFromLog`), encoding of non-ASCII titles. + +## How to work + +* Start with `git diff` (or the range you were given) and read the full functions around each hunk, not just the hunk. +* Run the unit tests (`python3 -m unittest discover -s tests -p 'test_*.py'`) and, if `tests/replay/out/baseline` + exists and the change touches matching/output code, confirm a replay comparison was produced; report its numbers. +* Verify each finding by reading the code; do not speculate. If you cannot confirm, say so and mark it PLAUSIBLE. + +## Output + +Findings ranked by severity, each with: file:line, one-sentence defect, a concrete failing input → wrong output, and +the minimal fix. Then a short list of what you verified and found correct. No praise, no restating the diff. diff --git a/.claude/agents/review-security.md b/.claude/agents/review-security.md new file mode 100644 index 0000000..11bff03 --- /dev/null +++ b/.claude/agents/review-security.md @@ -0,0 +1,55 @@ +--- +name: review-security +description: Security reviewer for booktree changes. Use after every code step, before committing, on the diff of the working tree or a commit range; also for reviewing third-party PRs before merging. +tools: Read, Grep, Glob, Bash +--- + +You review changes to booktree for security. booktree runs unattended (docker, non-root) against a download folder of +untrusted files, holds a private-tracker session cookie (MAM `mam_id`), calls the Audible and MAM HTTP APIs, and +writes into a media library that Audiobookshelf then parses. Report findings only; do not edit files. + +## Threat model + +* **Untrusted input:** file names and folder names under `source_path`; id3 tags read via ffprobe; every field of + Audible and MAM API responses; the contents of the cache directory and `cookies.pkl` (world-readable dirs on the + host); CSV log rows fed back in log mode (`fix.csv`); `--hints` JSON; mousehole `state.json`. +* **Assets:** the MAM session cookie (account ban / hit-and-run exposure if leaked or over-used), the seeded torrent + files (any write = data loss for the swarm and a tracker violation), the media library, the host filesystem + outside the mounted volumes. + +## What you check + +1. **Secrets.** The MAM session must never reach stdout, the CSV, the cache, exception messages, or `--json-log`. + Env/secret-file loading must not fall back to printing the value. Check that redaction covers `Cookie:` headers + in any debug output and that the config example files contain placeholders only. +2. **Path safety.** Every path built from metadata or file names (`getConfigTargetPath`, hardlink/copy targets, + OPF/cover destinations, cache files keyed by hash, calibre ingest path) must resolve under the intended root. + Check `sanitize_filename` is applied to each component, that `..`, leading `/`, and empty components cannot + escape, and that nothing is created under `source_path`. +3. **Deserialization and injection.** `pickle.load` of `cookies.pkl` from a shared directory is arbitrary code + execution if the file is replaced; flag any new pickle use and recommend a plain-text/JSON cookie store. XML/OPF: + all text nodes escaped or CDATA-safe (`]]>` inside CDATA). Shell: subprocess must use argument lists, never + `shell=True` with metadata. Regex: patterns built from metadata must be `re.escape`d (upstream `getAltTitle` and + `getCleanTitle` interpolate author/series names into regexes). JSON from MAM (`author_info` etc.) parsed with + `json.loads` only, sizes bounded. +4. **Network hygiene.** TLS verification on; timeouts on every request; MAM calls throttled (≥ 6 s) and bounded per + run; no retries that could hammer MAM on 429/403; errors never cached; `User-Agent` set. No new hosts contacted. +5. **Cache poisoning / TOCTOU.** Cache files are trusted on read: check that a malformed or attacker-written cache + entry cannot crash the run into a bad state or redirect a hardlink. Atomic writes for cache/state files. +6. **Supply chain.** New dependencies pinned in `requirements.txt`; GitHub Actions pinned to a major tag or SHA; + Dockerfile base pinned; nothing downloaded at runtime. +7. **Third-party PRs.** When reviewing an external PR (e.g. the mousehole integration), additionally look for + behaviour unrelated to the stated purpose, changed defaults, new network calls, and anything that reads files + outside the configured paths. + +## How to work + +* `git diff` (or the given range), then read the surrounding functions. Grep for every new `open(`, `os.`, + `shutil.`, `subprocess`, `pickle`, `requests`, `httpx`, `re.compile`, `print(` in the change. +* Run `bandit -q -r . -x ./tests -ll` and `pip-audit -r requirements.txt` if available and include real results. +* Confirm each finding by reading the code and constructing the malicious input; mark anything unconfirmed PLAUSIBLE. + +## Output + +Findings ranked by severity (critical / high / medium / low), each with: file:line, the vulnerability class, a +concrete attack input and consequence, and the minimal fix. Then a short list of checks performed that passed. diff --git a/.github/dependabot.yml b/.github/dependabot.yml new file mode 100644 index 0000000..691504b --- /dev/null +++ b/.github/dependabot.yml @@ -0,0 +1,11 @@ +version: 2 +updates: + - package-ecosystem: pip + directory: / + schedule: { interval: weekly } + - package-ecosystem: github-actions + directory: / + schedule: { interval: weekly } + - package-ecosystem: docker + directory: / + schedule: { interval: weekly } diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..1d490e0 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,91 @@ +name: CI + +on: + push: + branches: [ main ] + tags: [ 'v*' ] + pull_request: + workflow_dispatch: + +permissions: + contents: read + +jobs: + checks: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: { fetch-depth: 0 } + - uses: actions/setup-python@v5 + with: { python-version: '3.12' } + - name: Install + run: | + python -m pip install --upgrade pip + pip install -r requirements.txt ruff bandit pip-audit + - name: Lint + run: ruff check . + - name: Unit tests + run: python -m unittest discover -s tests -p 'test_*.py' -v + - name: Static security scan (bandit) + # high severity gates the build now; the two inherited medium findings (pickle load of cookies.pkl in + # myx_mam.py) are removed with the secrets/cookie-store work, after which this tightens to -ll + run: bandit -q -r . -x ./tests -lll + - name: Dependency audit + run: pip-audit -r requirements.txt + - name: Secret scan (gitleaks) + uses: gitleaks/gitleaks-action@v2 + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + + image: + needs: checks + runs-on: ubuntu-latest + permissions: + contents: read + packages: write + steps: + - uses: actions/checkout@v4 + - uses: docker/setup-qemu-action@v3 + - uses: docker/setup-buildx-action@v3 + - name: Login to GHCR + if: github.event_name != 'pull_request' + uses: docker/login-action@v3 + with: + registry: ghcr.io + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + - name: Image metadata + id: meta + uses: docker/metadata-action@v5 + with: + images: ghcr.io/mancolt/booktree + tags: | + type=raw,value=latest,enable={{is_default_branch}} + type=sha,format=long + type=semver,pattern={{version}} + - name: Build (and push on main/tags) + uses: docker/build-push-action@v5 + with: + context: . + platforms: linux/amd64,linux/arm64 + push: ${{ github.event_name != 'pull_request' }} + load: ${{ github.event_name == 'pull_request' }} + provenance: true + sbom: true + build-args: | + UID=1000 + GID=1000 + USER=booktree + GROUPNAME=booktree + tags: ${{ steps.meta.outputs.tags }} + labels: ${{ steps.meta.outputs.labels }} + cache-from: type=gha + cache-to: type=gha,mode=max + - name: Scan image (Trivy) + if: github.event_name != 'pull_request' + uses: aquasecurity/trivy-action@0.28.0 + with: + image-ref: ghcr.io/mancolt/booktree:sha-${{ github.sha }} + severity: HIGH,CRITICAL + exit-code: '1' + ignore-unfixed: true diff --git a/.github/workflows/docker-image.yml b/.github/workflows/docker-image.yml deleted file mode 100644 index 68e9c6c..0000000 --- a/.github/workflows/docker-image.yml +++ /dev/null @@ -1,34 +0,0 @@ -name: Build and push Docker image to dockerhub - -on: - push: - branches: [ "stable" ] - -jobs: - - build: - runs-on: ubuntu-latest - - steps: - - name: Check out the repo - uses: actions/checkout@v4 - - - name: Login to dockerhub - uses: docker/login-action@v3 - with: - username: ${{ secrets.DOCKERHUB_USERNAME }} - password: ${{ secrets.DOCKERHUB_TOKEN }} - - - name: Build and push to dockerhub - uses: docker/build-push-action@v5 - with: - context: . - push: true - build-args: | - UID=1000 - GID=1000 - USER=booktree - GROUPNAME=booktree - tags: | - myxdvz/booktree:latest - diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..fce13b6 --- /dev/null +++ b/.gitignore @@ -0,0 +1,12 @@ +__pycache__/ +*.py[cod] +.venv/ +# replay output is large (stdout/OPF transcripts per book); the summary is committed separately +tests/replay/out/ +# never commit live configs, logs, caches or sessions +config/ +logs/ +__cache__/ +*.pkl +# site-specific operational notes and handoffs +docs/site/ diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..cc2546e --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,20 @@ +# Contributing + +Thanks for helping. A few rules keep this fork safe to run unattended on other people's libraries. + +**Compatibility.** The run-log CSV columns and order, the `True`/`False` strings, the log file name pattern and the +stdout phrases listed in `docs/FORK.md` are a public interface. Add columns or flags; do not rename or reorder. + +**Never write under the source path.** Downloads are often live torrents. No moves, renames, tag rewrites, or files +created next to the sources. Hardlink/copy into the media path only. `fixid3` stays off by default. + +**External services.** MAM calls are bounded per run and spaced ≥ 6 s; Audible/MAM results are cached with a TTL for +empty results; errors are never cached. Any new call site must go through the existing throttled helpers. + +**Secrets.** Never commit configs, logs, caches or cookie files (`.gitignore` covers them). Tests use synthetic +fixtures only. CI runs gitleaks and push protection is on. + +**Tests.** `python3 -m unittest discover -s tests -p 'test_*.py'` must pass. Behaviour changes to matching or output +need a replay comparison (`tests/replay/`) and the numbers in the PR description. + +**Style.** `ruff check .` clean. Small, single-purpose commits; explain *why* in the message. diff --git a/Dockerfile b/Dockerfile index f658c46..22c03cc 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,4 +1,4 @@ -FROM alpine:latest +FROM alpine:3.21 ARG UID ARG GID diff --git a/README.md b/README.md index 97672bd..cc43f80 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,10 @@ -# booktree +# booktree (maintained fork) + +> This is a maintained fork of [myxdvz/booktree](https://github.com/myxdvz/booktree). It keeps booktree's file layout and +> run-log format so existing setups keep working, and fixes the matching, caching and OPF defects listed in +> [docs/FORK.md](docs/FORK.md). Image: `ghcr.io/mancolt/booktree:latest`. Issues and pull requests are welcome; see +> [CONTRIBUTING.md](CONTRIBUTING.md) and [SECURITY.md](SECURITY.md). + Reorganize your audiobooks using ID3 or Audible metadata into a tree structure recommended and supported by media servers like Audibookshelf. The originals are untouched and will be hardlinked to their destination It does the following: diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000..db55338 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,12 @@ +# Security policy + +booktree runs unattended against a download folder, holds a private-tracker session cookie, and writes into a media +library. Reports about any of the following are in scope: writes outside the configured media path, anything that +touches files under the source path, credential leakage (config, logs, cache, stdout), unsafe deserialization, +injection through file names or metadata (paths, OPF/XML, shell), and dependency vulnerabilities. + +Please report privately through GitHub's "Report a vulnerability" (Security tab) rather than a public issue. You will +get an acknowledgement within a week. Fixes ship as a new image tag; the advisory is published once a fixed image exists. + +Hardening in this fork: the container runs as a non-root user; secrets can be supplied by environment variable or +secret file instead of the config file; CI runs gitleaks, bandit, pip-audit and a Trivy scan of the built image. diff --git a/docs/FORK.md b/docs/FORK.md new file mode 100644 index 0000000..1dd1059 --- /dev/null +++ b/docs/FORK.md @@ -0,0 +1,47 @@ +# About this fork + +A maintained fork of [myxdvz/booktree](https://github.com/myxdvz/booktree) (GPL-3.0). Upstream has been inactive since +March 2026 with open issues and PRs; this fork fixes the defects below while keeping booktree's outputs compatible +with existing consumers (Audiobookshelf, scripts that parse the run log). Image: `ghcr.io/mancolt/booktree`. + +## Compatibility promise (v1 output contract) + +Scripts built on upstream booktree keep working: + +* Run log `booktree_log_YYYYMMDDHHMMSS.csv`: same 46 columns in the same order, one row per file, `isMatched` / + `isHardLinked` as the strings `True` / `False`. New information is added as new columns or via `--json-log`, never by + renaming or reordering. +* stdout phrases stay unchanged: `Processing: …`, `Searching Audible for` with indented `asin:` `title:` `authors:` + `narrators:` `keywords:`, `Searching MAM for` with `TitleFilename:`, `Checking cache: <kind>/<sha256>`, + `Caching <hash> in File: …`, `Found N Audible match(es)`, `Creating Hardlinks for N matched books`, + `Hardlinking files for <title>` with `from` / `to`. +* Target layout `{author}/{title}` and `{author}/{series}/{series} #{part} - {title}` (configurable as before). +* Source files are never moved, renamed, re-tagged or written next to; hardlinking (or copying) into the media path is + the only file operation, and `fixid3` stays off by default. Downloads are frequently seeded torrents. +* MAM is queried at most a bounded number of times per run with ≥ 6 s between calls. + +## Defects addressed (roadmap) + +| # | defect | upstream ref | +|---|---|---| +| 1 | `metadata.opf` text is not XML-escaped; a publisher such as `Little, Brown & Company` makes Audiobookshelf discard the whole OPF, ASIN included | — | +| 2 | With an explicit ASIN in log mode the id3 title/author still vetoes the match; no way to pass external hints or use duration as evidence | — | +| 3 | Missing id3 → the whole filename is the search title (`Author - Title.m4b` scores 41 against its own MAM entry) | #26 | +| 4 | Empty Audible / MAM results and skeleton per-ASIN responses (`{asin, asset_details, is_vvab}`) are cached forever | #25 | +| 5 | MAM session duplicated in every config file; `title_patterns` contain `"\b"` JSON escapes that become backspaces; exit code is 0 on failure | — | +| 6 | `Series # - Title` folders: the log writes `seriesparts` as `Name part`, the log reader splits on `#` | #27 | +| 7 | mousehole cookie integration (PR #24, updated for mousehole's v2 state file) | #24 | + +## Regression replay + +`tests/replay/` replays historical run logs offline inside the image: each book is rebuilt from its CSV rows, the +real search-key construction, Audible/MAM ranking (from cached API responses, network disabled), target path and OPF +generation are executed, and the stdout transcript is captured per book. `compare.py` diffs two runs; a +compatibility-preserving change must report `NO DIFFERENCES` for stdout, cache keys and target paths apart from the +change under test. `baseline-summary.json` is the frozen result for the upstream code (256 runs, 1,576 books: +target path reproduced for 261/266 matched books; 69/266 OPF files not well-formed; two consecutive runs identical). + +``` +BOOKTREE_LOGS=/path/to/logs BOOKTREE_CONFIG=/path/to/config tests/replay/run_replay.sh ghcr.io/mancolt/booktree:dev out/ +python3 tests/replay/compare.py tests/replay/out/baseline out/ +``` diff --git a/requirements.txt b/requirements.txt index bfbea88..9d258e4 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,5 +1,7 @@ httpx==0.27.0 langcodes==3.4.0 pathvalidate==3.2.0 -Requests==2.32.3 +Requests==2.33.0 thefuzz==0.22.1 +# pinned: thefuzz scores come from rapidfuzz; a silent bump would move match rates against the corpus baseline +rapidfuzz==3.14.6 diff --git a/ruff.toml b/ruff.toml new file mode 100644 index 0000000..4a321a3 --- /dev/null +++ b/ruff.toml @@ -0,0 +1,12 @@ +line-length = 140 +target-version = "py312" + +[lint] +select = ["E9", "F63", "F7", "F82", "F401", "F811", "F841", "B006", "B008", "S102", "S301", "S307", "S602", "S605"] + +[lint.per-file-ignores] +# Inherited upstream modules are exempt from the style rules until they are refactored; new code is not. +"booktree.py" = ["F401", "F841"] +"myx_*.py" = ["F401", "F841"] +# S301: pickle load of cookies.pkl — removed together with the secrets/cookie-store work (roadmap item 5) +"myx_mam.py" = ["F401", "F841", "S301"] diff --git a/tests/__init__.py b/tests/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/replay/baseline-summary.json b/tests/replay/baseline-summary.json new file mode 100644 index 0000000..4bd1ab1 --- /dev/null +++ b/tests/replay/baseline-summary.json @@ -0,0 +1,37 @@ +{ + "runs": 256, + "rows": 23892, + "books": 1576, + "logged_matched_books": 266, + "logged_hardlinked_rows": 2776, + "passes": { + "pass1": { + "audible_cache_hits": 870, + "adb_agree": 846, + "adb_disagree": 24, + "mam_cache_hits": 1426, + "matchFound": 590, + "exceptions": 0 + }, + "pass2": { + "audible_cache_hits": 963, + "adb_agree": 910, + "adb_disagree": 53, + "mam_cache_hits": 0, + "matchFound": 126, + "exceptions": 0 + } + }, + "target_path": { + "compared": 266, + "agree": 261 + }, + "opf": { + "generated": 266, + "bare_amp": 69, + "not_well_formed": 69 + }, + "stdout_sha_all": "ae625fd7cf79cdaf9d768dedc3f54189a8983545b6ef2ee9b661b9328fae3919", + "opf_sha_all": "a07477bb215500f17617def3c3a98ac0d872f131940faa3b64aeb18c3f74b634", + "seconds": 4.6 +} \ No newline at end of file diff --git a/tests/replay/compare.py b/tests/replay/compare.py new file mode 100755 index 0000000..7e215d7 --- /dev/null +++ b/tests/replay/compare.py @@ -0,0 +1,63 @@ +#!/usr/bin/env python3 +"""Diff two replay outputs (books.jsonl) book by book. + +usage: compare.py <baseline-dir> <candidate-dir> [--show N] +Reports, per pass, how many books changed: stdout transcript, best Audible ASIN, best MAM ASIN, +matchFound, cache keys (search-key construction), plus target path and OPF changes. Exit 0 always; +the numbers are the verdict — a byte-compat step must show 0 changes everywhere, an A/B step shows +what it changed. +""" +import argparse, json, os + + +def load(d): + out = {} + with open(os.path.join(d, "books.jsonl"), encoding="utf-8") as fh: + for line in fh: + r = json.loads(line) + out[(r["run"], r["book_hash"])] = r + return out + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("base"); ap.add_argument("cand"); ap.add_argument("--show", type=int, default=10) + a = ap.parse_args() + A, B = load(a.base), load(a.cand) + keys = sorted(set(A) | set(B)) + counts = {} + examples = {} + + def bump(cat, key, detail): + counts[cat] = counts.get(cat, 0) + 1 + examples.setdefault(cat, []).append((key, detail)) + + for k in keys: + ra, rb = A.get(k), B.get(k) + if ra is None or rb is None: + bump("books_only_in_one_side", k, "baseline" if rb is None else "candidate"); continue + for tag in sorted(set(ra["passes"]) | set(rb["passes"])): + pa, pb = ra["passes"].get(tag, {}), rb["passes"].get(tag, {}) + for field in ("stdout_sha", "best_adb_asin", "best_mam_asin", "matchFound", "cache_keys", "search_title_used", "audibleMatches", "mamMatches"): + if pa.get(field) != pb.get(field): + bump(f"{tag}.{field}", k, f"{ra['book']!r}: {pa.get(field)!r} -> {pb.get(field)!r}" if field != "stdout_sha" else ra["book"]) + if pa.get("matchFound") != pb.get("matchFound"): + bump(f"{tag}.match_" + ("gained" if pb.get("matchFound") else "lost"), k, ra["book"]) + if ra["target_path"].get("computed") != rb["target_path"].get("computed"): + bump("target_path", k, f"{ra['target_path'].get('computed')!r} -> {rb['target_path'].get('computed')!r}") + oa, ob = ra.get("opf") or {}, rb.get("opf") or {} + if oa.get("sha") != ob.get("sha"): + bump("opf_text", k, ra["book"]) + if oa.get("well_formed") != ob.get("well_formed"): + bump("opf_well_formed_" + ("fixed" if ob.get("well_formed") else "broken"), k, ra["book"]) + print(f"books compared: {len(keys)} (baseline {len(A)}, candidate {len(B)})") + if not counts: + print("NO DIFFERENCES"); return + for cat in sorted(counts): + print(f"\n== {cat}: {counts[cat]}") + for key, detail in examples[cat][:a.show]: + print(f" {key[0]} {detail}") + + +if __name__ == "__main__": + main() diff --git a/tests/replay/replay_corpus.py b/tests/replay/replay_corpus.py new file mode 100644 index 0000000..5715858 --- /dev/null +++ b/tests/replay/replay_corpus.py @@ -0,0 +1,310 @@ +#!/usr/bin/env python3 +"""Offline regression replay of booktree over the historical run-log corpus. + +For every historical run log (booktree_log_*.csv) the rows are regrouped into books exactly as +buildTreeFromLog does (one book per (row-index, book) key), the id3 Book is rebuilt from the id3-* +columns, and the REAL booktree code paths are executed with the network disabled: + + * MAMBook.getAudibleBooks -> search-key construction (getAltTitle, cleanseTitle, optimizeKeys), + cache-key hashing, ranking of the cached Audible response + * MAMBook.getMAMBooks -> MAM search-string construction, cache-key hashing, ranking of the + cached MAM response (snatched-only filter) + * BookFile.getConfigTargetPath -> the hardlink target ("paths" column) for the logged best match + * myx_utilities.createOPF -> the metadata.opf text for the logged best match + * every stdout line the above print (the phrases downstream consumers scrape) + +Nothing is written outside --out. Cache and logs are only read. Output: + <out>/books.jsonl one record per (run, book) with keys, hashes, replayed vs logged results + <out>/stdout/<run>/<bookhash>.txt the captured stdout transcript per book + <out>/opf/<run>/<bookhash>.opf the generated OPF per matched book + <out>/summary.json aggregate counts (this is what a later run is compared against) + +Run it inside the booktree image with the repo mounted over /booktree (see run_baseline.sh). +""" +import argparse, contextlib, csv, glob, hashlib, io, json, os, sys, tempfile, time +from types import SimpleNamespace + +sys.path.insert(0, os.getcwd()) # booktree modules live in the cwd (/booktree) +import myx_args, myx_classes, myx_utilities, myx_mam # noqa: E402 + + +# ---------------------------------------------------------------- network kill-switch +class _Offline(Exception): + pass + + +class OfflineClient: + """Stands in for httpx: any request raises, so getAudibleBook falls through to its except branch + (prints 'Error searching audible: ...') and never calls cacheMe.""" + def get(self, *a, **k): + raise _Offline("offline replay: network disabled") + + +class _OfflineSession: + def __init__(self): + self.headers = {} + self.cookies = None + def get(self, *a, **k): + raise _Offline("offline replay: network disabled") + def post(self, *a, **k): + raise _Offline("offline replay: network disabled") + + +myx_mam.requests = SimpleNamespace(Session=_OfflineSession) # searchMAM builds requests.Session() + + +# ---------------------------------------------------------------- helpers +def load_cfg(path, cache_path, log_path): + params = SimpleNamespace(config_file=path, dry_run=None, verbose=True, no_cache=None, no_opf=None, + multibook=None, ebooks=None, fixid3=None, add_narrators=None) + cfg = myx_args.Config(params) + cfg._data["Config"]["cache_path"] = cache_path + cfg._data["Config"]["log_path"] = log_path + cfg._data["Config"]["session"] = "" # never needed offline; never log it + return cfg + + +def book_from_row(row, ns): + b = myx_classes.Book(asin=str(row[f"{ns}asin"]), title=str(row[f"{ns}title"]), subtitle=row[f"{ns}subtitle"], + publisher=row[f"{ns}publisher"], length=row[f"{ns}length"], duration=row[f"{ns}duration"], + language=row[f"{ns}language"] or "english") + b.setAuthors(row[f"{ns}authors"] or "") + b.setNarrators(row[f"{ns}narrators"] or "") + set_series_from_log(b, row[f"{ns}series"] or "", row[f"{ns}seriesparts"] or "") + return b + + +def set_series_from_log(book, series_col, seriesparts_col): + """Rebuild Series objects from the log columns. + + The CSV writes `series` = getSeries() (names, comma-joined) and `seriesparts` = getSeriesParts() + = "<name> <separator><part>" where the separator is "" for Audible/MAM books, i.e. "The Witcher 1". + Upstream's log-mode reader (Book.setSeries) splits on '#', so it turns that into + Series(name="The Witcher 1", part="") -- the origin of the `Series # - Title` folders (upstream + issue #27 in log mode). The replay wants the metadata as the ORIGINAL hybrid run saw it, so we + strip the known series name off the front of the seriesparts value to recover the part. + Falls back to upstream's setSeries when the columns cannot be aligned.""" + names = [n.strip() for n in series_col.split(",") if n.strip()] + parts = seriesparts_col.strip() + # seriesparts went through cleanseAuthor (dots removed, whitespace collapsed); series through cleanseSeries + key = myx_utilities.cleanseAuthor(names[0]) if len(names) == 1 else None + if key is not None and parts.startswith(key): + book.series.append(myx_classes.Series(names[0], parts[len(key):].strip().lstrip("#").strip())) + elif not names and parts: + book.setSeries(parts) + elif names: + # multiple series: best effort, positional split is ambiguous -> keep upstream behaviour + book.setSeries(parts) + + +def sha(s): + return hashlib.sha256(s.encode("utf-8", "surrogatepass")).hexdigest() + + +def group_run(path): + """Regroup one log exactly like buildTreeFromLog: key = hash(f'{i}-{book}') with i the 1-based row index.""" + books = {} + order = [] + with open(path, newline="", errors="ignore", encoding="utf-8") as fh: + reader = csv.DictReader(fh, fieldnames=list(myx_utilities.getLogHeaders().keys())) + i = 1 + for row in reader: + if i > 1: + f = str(row["file"]) + bf = myx_classes.BookFile(f, f, str(row["sourcePath"]), str(row["mediaPath"]), + isHardlinked=(str(row["isHardLinked"]).lower() == "true")) + bf.ffprobeBook = book_from_row(row, "id3-") + bf.isMatched = (str(row["isMatched"]).lower() == "true") + # NB: upstream keys on the row index, so every row is its own MAMBook in log mode; for a + # faithful replay of the hybrid run that produced the log we regroup by the 'book' key + # (folder or filename), which is what buildTreeFromHybridSources did. + key = str(row["book"]) + if key not in books: + mb = myx_classes.MAMBook(key) + mb.metadata = str(row["metadatasource"]) + mb.paths = str(row["paths"]) + mb.isMatched = bf.isMatched + mb.ffprobeBook = bf.ffprobeBook + mb._rows = [] + books[key] = mb + order.append(key) + books[key].files.append(bf) + books[key]._rows.append(row) + i += 1 + return [books[k] for k in order] + + +def replay_book(mb, cfgs, out_dirs, run): + rows = mb._rows + r0 = rows[0] + rec = { + "run": run, "book": mb.name, "book_hash": sha(mb.name), "files": len(rows), + "logged": { + "isMatched": r0["isMatched"], "isHardLinked": [r["isHardLinked"] for r in rows], + "metadatasource": r0["metadatasource"], "mamCount": r0["mamCount"], + "audibleMatchCount": r0["audibleMatchCount"], "adb_asin": r0["adb-asin"], "mam_asin": r0["mam-asin"], + "adb_matchRate": r0["adb-matchRate"], "mam_matchRate": r0["mam-matchRate"], "paths": r0["paths"], + }, + "id3": {"asin": r0["id3-asin"], "title": r0["id3-title"], "authors": r0["id3-authors"], + "narrators": r0["id3-narrators"], "duration": r0["id3-duration"]}, + "passes": {}, + } + for tag, cfg in cfgs.items(): + buf = io.StringIO() + # fresh MAMBook per pass: getAudibleBooks mutates book.title via getAltTitle + b = myx_classes.MAMBook(mb.name) + b.metadata = "id3" + for bf in mb.files: + nbf = myx_classes.BookFile(bf.file, bf.fullPath, bf.sourcePath, bf.mediaPath) + nbf.ffprobeBook = book_from_row(next(r for r in rows if r["file"] == bf.file), "id3-") + b.files.append(nbf) + b.ffprobeBook = b.files[0].ffprobeBook + metadata = cfg.get("Config/metadata") + p = {"metadata": metadata} + with contextlib.redirect_stdout(buf): + print(f"Processing: {b.name}...") + try: + if "mam" in metadata: + b.getMAMBooks(cfg, b.files[0]) + if b.bestMAMMatch is not None: + b.metadata = "mam" + if "audible" in metadata: + src = b.bestMAMMatch if (b.bestMAMMatch is not None and b.bestMAMMatch.language.lower() != "english") else None + if metadata == "mam-audible" and src is None and not myx_utilities.isMultiBookCollection(b.files[0].file): + src = b.bestMAMMatch + if src is None: + src = b.ffprobeBook + b.getAudibleBooks(OfflineClient(), src, cfg) + if b.bestAudibleMatch is not None: + b.metadata = "audible" + print(f"Found {len(b.mamMatches)} MAM matches, {len(b.audibleMatches)} Audible Matches") + myx_utilities.printDivider() + except _Offline: + pass + except Exception as e: # replay must never abort on one book + print(f"REPLAY-EXCEPTION {type(e).__name__}: {e}") + p["exception"] = f"{type(e).__name__}: {e}" + text = buf.getvalue() + cache_refs = [l.split("Checking cache: ", 1)[1].rstrip(".") for l in text.splitlines() if l.startswith("Checking cache: ")] + hits = [l for l in text.splitlines() if l.startswith("Retrieving ")] + p.update({ + "cache_keys": cache_refs, + "audible_cache_hit": any(h.endswith("from audible") for h in hits), + "mam_cache_hit": any(k.startswith("mam/") and os.path.exists(os.path.join(cfg.get("Config/cache_path"), "__cache__", k)) for k in cache_refs), + "audibleMatches": len(b.audibleMatches or []), "mamMatches": len(b.mamMatches or []), + "best_adb_asin": b.bestAudibleMatch.asin if b.bestAudibleMatch else "", + "best_adb_rate": b.bestAudibleMatch.matchRate if b.bestAudibleMatch else "", + "best_mam_asin": b.bestMAMMatch.asin if b.bestMAMMatch else "", + "matchFound": b.matchFound(), "search_title_used": b.ffprobeBook.title, + "stdout_sha": sha(text), "stdout_lines": len(text.splitlines()), + }) + p["adb_agrees_with_log"] = (p["best_adb_asin"] == r0["adb-asin"]) if p["audible_cache_hit"] else None + d = os.path.join(out_dirs["stdout"], run); os.makedirs(d, exist_ok=True) + with open(os.path.join(d, f"{rec['book_hash']}.{tag}.txt"), "w", encoding="utf-8", errors="surrogateescape") as fh: + fh.write(text) + rec["passes"][tag] = p + + # ---- deterministic parts from the LOGGED best match: target path + OPF + cfg = next(iter(cfgs.values())) + src = r0["metadatasource"] + meta = None + if src == "audible" and r0["adb-asin"]: + meta = book_from_row(r0, "adb-") + elif src == "mam" and (r0["mam-title"] or r0["mam-asin"]): + meta = book_from_row(r0, "mam-") + rec["target_path"] = {"logged": r0["paths"], "computed": None, "agree": None} + rec["opf"] = None + if meta is not None: + try: + computed = mb.files[0].getConfigTargetPath(cfg, meta) + rec["target_path"].update(computed=computed, agree=(computed == r0["paths"])) + except Exception as e: + rec["target_path"]["computed"] = f"EXC {type(e).__name__}: {e}" + with tempfile.TemporaryDirectory() as td, contextlib.redirect_stdout(io.StringIO()): + myx_utilities.createOPF(meta, td) + opf_path = os.path.join(td, "metadata.opf") + if os.path.exists(opf_path): + opf = open(opf_path, encoding="utf-8").read() + d = os.path.join(out_dirs["opf"], run); os.makedirs(d, exist_ok=True) + with open(os.path.join(d, f"{rec['book_hash']}.opf"), "w", encoding="utf-8") as fh: + fh.write(opf) + rec["opf"] = {"sha": sha(opf), "bytes": len(opf), + "has_bare_amp": ("&" in opf and "&" not in opf), + "well_formed": xml_ok(opf)} + return rec + + +def xml_ok(text): + import xml.etree.ElementTree as ET + try: + ET.fromstring(text.encode("utf-8")) + return True + except ET.ParseError: + return False + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--logs", default="/logs") + ap.add_argument("--cache", default="/Config", help="dir that contains __cache__/") + ap.add_argument("--config", action="append", required=True, help="tag=path, e.g. pass1=/Config/config.json") + ap.add_argument("--out", required=True) + ap.add_argument("--limit", type=int, default=0) + a = ap.parse_args() + + out_dirs = {k: os.path.join(a.out, k) for k in ("stdout", "opf")} + for d in out_dirs.values(): + os.makedirs(d, exist_ok=True) + cfgs = {} + for spec in a.config: + tag, path = spec.split("=", 1) + cfgs[tag] = load_cfg(path, a.cache, a.logs) + + logs = sorted(glob.glob(os.path.join(a.logs, "booktree_log_*.csv"))) + if a.limit: + logs = logs[-a.limit:] + t0 = time.time() + summary = {"runs": 0, "rows": 0, "books": 0, "logged_matched_books": 0, "logged_hardlinked_rows": 0, + "passes": {t: {"audible_cache_hits": 0, "adb_agree": 0, "adb_disagree": 0, "mam_cache_hits": 0, + "matchFound": 0, "exceptions": 0} for t in cfgs}, + "target_path": {"compared": 0, "agree": 0}, "opf": {"generated": 0, "bare_amp": 0, "not_well_formed": 0}, + "stdout_sha_all": hashlib.sha256(), "opf_sha_all": hashlib.sha256()} + with open(os.path.join(a.out, "books.jsonl"), "w", encoding="utf-8") as jl: + for lp in logs: + run = os.path.basename(lp)[len("booktree_log_"):-4] + books = group_run(lp) + summary["runs"] += 1 + for mb in books: + rec = replay_book(mb, cfgs, out_dirs, run) + summary["rows"] += rec["files"]; summary["books"] += 1 + summary["logged_matched_books"] += rec["logged"]["isMatched"] == "True" + summary["logged_hardlinked_rows"] += sum(h == "True" for h in rec["logged"]["isHardLinked"]) + for t, p in rec["passes"].items(): + s = summary["passes"][t] + s["audible_cache_hits"] += p["audible_cache_hit"] + s["mam_cache_hits"] += p["mam_cache_hit"] + s["matchFound"] += p["matchFound"] + s["exceptions"] += "exception" in p + if p["adb_agrees_with_log"] is True: s["adb_agree"] += 1 + if p["adb_agrees_with_log"] is False: s["adb_disagree"] += 1 + summary["stdout_sha_all"].update(p["stdout_sha"].encode()) + if rec["target_path"]["agree"] is not None: + summary["target_path"]["compared"] += 1 + summary["target_path"]["agree"] += rec["target_path"]["agree"] + if rec["opf"]: + summary["opf"]["generated"] += 1 + summary["opf"]["bare_amp"] += rec["opf"]["has_bare_amp"] + summary["opf"]["not_well_formed"] += not rec["opf"]["well_formed"] + summary["opf_sha_all"].update(rec["opf"]["sha"].encode()) + jl.write(json.dumps(rec, ensure_ascii=False) + "\n") + print(f"[{summary['runs']}/{len(logs)}] {run}: {len(books)} books", file=sys.stderr) + summary["stdout_sha_all"] = summary["stdout_sha_all"].hexdigest() + summary["opf_sha_all"] = summary["opf_sha_all"].hexdigest() + summary["seconds"] = round(time.time() - t0, 1) + with open(os.path.join(a.out, "summary.json"), "w") as fh: + json.dump(summary, fh, indent=1) + print(json.dumps(summary, indent=1)) + + +if __name__ == "__main__": + main() diff --git a/tests/replay/run_replay.sh b/tests/replay/run_replay.sh new file mode 100755 index 0000000..68df357 --- /dev/null +++ b/tests/replay/run_replay.sh @@ -0,0 +1,21 @@ +#!/bin/bash +# Replay the historical corpus offline inside a booktree image. +# usage: tests/replay/run_replay.sh <image> <out-dir> [--limit N] +# Mounts: repo -> /booktree (ro, overrides the image's code), logs -> /logs (ro), a *copy* of the +# cache + session-less configs -> /Config (ro). Nothing on the live host is written. +set -euo pipefail +IMAGE=${1:?image}; OUT=${2:?out dir}; shift 2 +REPO=$(cd "$(dirname "$0")/../.." && pwd) +LOGS=${BOOKTREE_LOGS:?set BOOKTREE_LOGS to the directory holding booktree_log_*.csv} +CONFIG=${BOOKTREE_CONFIG:?set BOOKTREE_CONFIG to the directory holding __cache__/ and the *.json configs} +STAGE=${BOOKTREE_STAGE:-$(mktemp -d)} +mkdir -p "$OUT" "$STAGE/Config" +# snapshot cache + configs (session blanked) so the container never sees the live dirs read-write +rsync -a --delete "$CONFIG/__cache__" "$STAGE/Config/" +for c in "$CONFIG"/*.json; do + sed -E 's/("session"[[:space:]]*:[[:space:]]*")[^"]*"/\1"/' "$c" > "$STAGE/Config/$(basename "$c")" +done +docker run --rm --user root -e PYTHONDONTWRITEBYTECODE=1 -w /booktree \ + -v "$REPO:/booktree:ro" -v "$LOGS:/logs:ro" -v "$STAGE/Config:/Config:ro" -v "$OUT:/out" \ + "$IMAGE" /venv/bin/python3 /booktree/tests/replay/replay_corpus.py \ + --config pass1=/Config/config.json --config pass2=/Config/config-audible.json --out /out "$@" diff --git a/tests/test_output_contract.py b/tests/test_output_contract.py new file mode 100644 index 0000000..9136a6e --- /dev/null +++ b/tests/test_output_contract.py @@ -0,0 +1,30 @@ +"""The run-log schema is a public interface (docs/FORK.md). These tests pin it.""" +import unittest + +import myx_utilities + +EXPECTED_COLUMNS = [ + "book", "file", "paths", "isMatched", "isHardLinked", "mamCount", "audibleMatchCount", "metadatasource", + "id3-matchRate", "id3-asin", "id3-title", "id3-subtitle", "id3-publisher", "id3-length", "id3-duration", + "id3-series", "id3-authors", "id3-narrators", "id3-seriesparts", "id3-language", + "mam-matchRate", "mam-asin", "mam-title", "mam-subtitle", "mam-publisher", "mam-length", "mam-duration", + "mam-series", "mam-authors", "mam-narrators", "mam-seriesparts", "mam-language", + "adb-matchRate", "adb-asin", "adb-title", "adb-subtitle", "adb-publisher", "adb-length", "adb-duration", + "adb-series", "adb-authors", "adb-narrators", "adb-seriesparts", "adb-language", + "sourcePath", "mediaPath", +] + + +class LogSchemaTest(unittest.TestCase): + def test_log_headers_are_the_46_v1_columns_in_order(self): + self.assertEqual(list(myx_utilities.getLogHeaders().keys()), EXPECTED_COLUMNS) + self.assertEqual(len(EXPECTED_COLUMNS), 46) + + def test_booleans_are_logged_as_python_true_false_strings(self): + # consumers compare the CSV cell to the literal strings "True"/"False" + self.assertEqual(str(True), "True") + self.assertEqual(str(False), "False") + + +if __name__ == "__main__": + unittest.main() From b44bf573f12738c26250365e7bbc13763b9dc7b7 Mon Sep 17 00:00:00 2001 From: Mancolt <mancolt86@gmail.com> Date: Mon, 14 Sep 2026 20:50:02 -0400 Subject: [PATCH 2/3] ci: fix action pins (trivy-action tags are v0.x), drop load for multi-arch PR builds, authenticate Trivy pull Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JdymXZSW4aRWMFREPAFzkZ --- .github/workflows/ci.yml | 26 +++++++++++++++----------- 1 file changed, 15 insertions(+), 11 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 1d490e0..9be0870 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -14,9 +14,9 @@ jobs: checks: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v5 with: { fetch-depth: 0 } - - uses: actions/setup-python@v5 + - uses: actions/setup-python@v6 with: { python-version: '3.12' } - name: Install run: | @@ -33,7 +33,7 @@ jobs: - name: Dependency audit run: pip-audit -r requirements.txt - name: Secret scan (gitleaks) - uses: gitleaks/gitleaks-action@v2 + uses: gitleaks/gitleaks-action@v3 env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} @@ -44,19 +44,19 @@ jobs: contents: read packages: write steps: - - uses: actions/checkout@v4 - - uses: docker/setup-qemu-action@v3 - - uses: docker/setup-buildx-action@v3 + - uses: actions/checkout@v5 + - uses: docker/setup-qemu-action@v4 + - uses: docker/setup-buildx-action@v4 - name: Login to GHCR if: github.event_name != 'pull_request' - uses: docker/login-action@v3 + uses: docker/login-action@v4 with: registry: ghcr.io username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} - name: Image metadata id: meta - uses: docker/metadata-action@v5 + uses: docker/metadata-action@v6 with: images: ghcr.io/mancolt/booktree tags: | @@ -64,12 +64,12 @@ jobs: type=sha,format=long type=semver,pattern={{version}} - name: Build (and push on main/tags) - uses: docker/build-push-action@v5 + uses: docker/build-push-action@v7 with: context: . platforms: linux/amd64,linux/arm64 + # pull requests build both platforms into the cache only (multi-arch images cannot be loaded locally) push: ${{ github.event_name != 'pull_request' }} - load: ${{ github.event_name == 'pull_request' }} provenance: true sbom: true build-args: | @@ -83,7 +83,11 @@ jobs: cache-to: type=gha,mode=max - name: Scan image (Trivy) if: github.event_name != 'pull_request' - uses: aquasecurity/trivy-action@0.28.0 + uses: aquasecurity/trivy-action@v0.36.0 + env: + # the package may be private until visibility is flipped; pull with the workflow token + TRIVY_USERNAME: ${{ github.actor }} + TRIVY_PASSWORD: ${{ secrets.GITHUB_TOKEN }} with: image-ref: ghcr.io/mancolt/booktree:sha-${{ github.sha }} severity: HIGH,CRITICAL From 1a1ddccba1f7f990c7ef166debf9b168daced008 Mon Sep 17 00:00:00 2001 From: Mancolt <mancolt86@gmail.com> Date: Mon, 14 Sep 2026 20:57:01 -0400 Subject: [PATCH 3/3] docker: drop system pip/setuptools, remove venv pip after install Trivy flagged Alpine's py3-setuptools 70.3.0 (CVE-2025-47273) and py3-msgpack (GHSA-6v7p-g79w-8964), both pulled in only by 'apk add py3-pip'. The venv seeds its own pip via ensurepip, so the system packages are unnecessary; the venv's pip is removed after installing requirements so the runtime image carries no installer. Also 'apk upgrade' the base and pin build args to their CI defaults. Runtime unchanged: Python 3.12, same dependency versions; corpus replay identical to baseline. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JdymXZSW4aRWMFREPAFzkZ --- Dockerfile | 38 ++++++++++++++++++-------------------- 1 file changed, 18 insertions(+), 20 deletions(-) diff --git a/Dockerfile b/Dockerfile index 22c03cc..4a8823f 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,31 +1,29 @@ FROM alpine:3.21 -ARG UID -ARG GID -ARG USER -ARG GROUPNAME -ENV USER=${USER} -ENV GROUPNAME=${GROUPNAME} -ENV UID=${UID} -ENV GID=${GID} +ARG UID=1000 +ARG GID=1000 +ARG USER=booktree +ARG GROUPNAME=booktree +ENV USER=${USER} GROUPNAME=${GROUPNAME} UID=${UID} GID=${GID} COPY . /booktree/ -WORKDIR /booktree +WORKDIR /booktree RUN echo "**** installing system packages ****" \ && apk update \ - && apk add --update --no-cache python3 py3-pip \ - && apk add py3-pip \ - && apk add ffmpeg \ - && ln -sf python3 /usr/bin/python \ - && mkdir -p /venv \ - && python -m venv /venv \ - && source /venv/bin/activate \ - && pip install --no-cache-dir --requirement requirements.txt \ - && pip install --upgrade pip \ + && apk upgrade --no-cache \ + # python3 ships ensurepip, so the venv gets its own pip; no system pip/setuptools are installed + && apk add --no-cache python3 ffmpeg \ + && ln -sf python3 /usr/bin/python \ + && python3 -m venv /venv \ + && /venv/bin/pip install --no-cache-dir --upgrade pip \ + && /venv/bin/pip install --no-cache-dir --requirement requirements.txt \ + # the runtime image needs no installer; removing it also removes the scanner surface + && /venv/bin/pip uninstall -y pip \ + && rm -rf /root/.cache \ && if getent passwd ${UID} >/dev/null; then deluser $(getent passwd ${UID} | cut -d: -f1); fi \ && if getent group ${GID} >/dev/null; then delgroup $(getent group ${GID} | cut -d: -f1); fi \ - && addgroup --system --gid ${GID} ${GROUPNAME} \ - && adduser --system --uid ${UID} --disabled-password --gecos "" --ingroup ${GROUPNAME} --no-create-home ${USER} \ + && addgroup -S -g ${GID} ${GROUPNAME} \ + && adduser -S -u ${UID} -G ${GROUPNAME} -H -D ${USER} \ && chown -R ${UID}:${GID} /booktree USER ${USER}