v6.6.0

NEWS.md

@@ -1,5 +1,81 @@
 # 📰 Picopt News
 
+## v6.6.0
+
+### Fixes
+
+- Files are now replaced atomically: a crash, kill, or full disk mid-write can
+  no longer destroy the original image.
+- Archive members whose optimization fails are kept in the repacked archive;
+  previously they were silently dropped.
+- Nested archives (e.g. a CBZ inside a ZIP) now optimize; previously they always
+  errored.
+- Archives keep their comments and member order through repack; tar symlinks and
+  hardlinks survive; 7z member times are preserved; pre-1980 member times no
+  longer crash conversion; legacy non-UTF8 zip member names are repaired.
+- Archive conversions compress each member by content; converting a rar, tar, or
+  7z no longer leaves every member stored uncompressed.
+- Files that fail to optimize are retried on the next timestamped run;
+  previously directory timestamps covered them and they were skipped forever.
+- One corrupt file inside an archive (e.g. a decompression bomb) no longer
+  aborts the entire run.
+- Boolean options set in config files or environment variables are no longer
+  ignored.
+- Requested archive conversions (e.g. CBR to CBZ) now happen even when the
+  contents are already optimized.
+- Lossless WebPs with large metadata inside archives are no longer misdetected
+  as lossy.
+- Owner-password-restricted PDFs keep their encryption; optimized PDF images
+  keep their color transform; more signed PDFs are detected and refused.
+- svgo no longer strips viewBox, and keeps title/desc metadata unless
+  `--strip-metadata` is given.
+- Conversions discarded for being bigger (and dry runs) are reported as skips,
+  not errors.
+- `--preserve` no longer fails as a non-root user when it cannot change
+  ownership; it still restores permissions and modification time.
+- Temporary files and animated-WebP frame directories are cleaned up when a tool
+  fails or a run is cancelled, instead of leaking to disk.
+- `picopt doctor` no longer reports missing tools and exits nonzero on healthy
+  installs.
+- Symlink loops, unreadable directories, and FIFOs no longer hang or crash the
+  walk.
+- Bad `--memory-limit` and `--after` values abort with clean error messages;
+  timezone-aware `--after` values are honored; lowercase format lists are
+  accepted for `-x` and `-c`.
+- `-I`/`--no-default-ignores` without `-i` no longer aborts at startup.
+- Existing timestamps are invalidated once after upgrading because the timestamp
+  config check gained new keys; the first `-t` run re-examines already-optimized
+  trees.
+
+### Features
+
+- Per-directory `.picopt.yaml` config files: pin any setting to a directory
+  tree. Deeper directories win; environment variables and the command line still
+  override. Editing one re-processes its tree on timestamped runs.
+- Write your invoked options to config files: `-w` to the user config, `-W` to a
+  `.picopt.yaml` in each target directory, `--write-config-file PATH` to an
+  explicit path.
+- Memory-aware scheduling: picopt now estimates the peak memory of large
+  archives and limits how many run at once so big libraries (e.g. multi-GB comic
+  archives) no longer exhaust RAM and get the process OOM-killed. Tune the
+  budget with the new `--memory-limit` option (e.g. `--memory-limit 8G`), which
+  reads as an approximate peak-memory target; the default is two-thirds of
+  detected RAM.
+- Incremental archive re-optimization: when a timestamped archive changes,
+  members older than its timestamp are skipped instead of re-optimized; `-E`
+  disables the member check. Timestamp files found inside archives are consumed
+  and removed on repack.
+
+### Performance
+
+- Archive member format detection runs in parallel workers instead of
+  serializing the scheduler.
+- Solid 7z archives extract in one pass instead of decompressing from the start
+  for every member.
+- The progress pre-scan is skipped in quiet mode, less image data is shipped
+  between processes, and scheduling under a memory limit no longer rescans the
+  whole queue on every completion.
+
 ## v6.5.2
 
 - Update to latest oxipng v10

README.md

@@ -110,15 +110,9 @@ When configured to convert GIFS to WebP, Animated GIFs are converted to WebP
 with the [gif2webp](https://developers.google.com/speed/webp/docs/gif2webp)
 binary if it exists. It is normally distributed as part of the webp package.
 
-#### Experimental option
-
-This experimental options is activated with an environment variable.
-
-Set `PICOPT_ENABLE_IMG2WEBP=1` to make picopt optimize animated WebPs and
-convert animated PNGs to webp with the
-[img2webp](https://developers.google.com/speed/webp/docs/img2webp) binary. In my
-experiments img2webp has performed worse than picopt's custom algorithm using
-PIL & cwebp
+Animated WebPs are optimized (and animated PNGs converted to WebP) with the
+[img2webp](https://developers.google.com/speed/webp/docs/img2webp) binary when
+it is available, falling back to picopt's internal algorithm using PIL & cwebp.
 
 ### SVG
 
@@ -243,6 +237,54 @@ Or you can install svgo with npm:
 npm install -G svgo
 ```
 
+## ⚙️ Configuration
+
+Picopt layers configuration from these sources, lowest to highest priority:
+packaged defaults, your user config file, a `-C CONFIG` file, per-directory
+`.picopt.yaml` files (see below), environment variables, and command line
+options. Every config file uses the same envelope:
+
+<!-- eslint-skip -->
+
+```yaml
+picopt:
+    recurse: true
+    convert_to: [WEBP, CBZ]
+```
+
+### Per-directory `.picopt.yaml` files
+
+A `.picopt.yaml` placed inside a target directory applies to that directory and
+everything under it. Files are discovered from each processed file's directory
+up to the command line target root (never above it), and deeper directories win
+over shallower ones. Environment variables and command line options still win
+over every directory file.
+
+<!-- eslint-skip -->
+
+```sh
+printf 'picopt:\n  convert_to: [CBZ]\n' > comics/.picopt.yaml
+picopt -rx CBR,CBZ .   # CBRs under comics/ convert; siblings don't
+```
+
+Any config key is accepted and validated, but run-scoped keys — `dry_run`,
+`list_only`, `timestamps`, `after`, `jobs`, `memory_limit`, `fail_fast`,
+`fail_fast_container`, `verbose`, and `paths` — are governed by the run-level
+value; setting them in a directory file has no per-directory effect. When
+timestamps (`-t`) are enabled, editing, adding, or removing any `.picopt.yaml`
+re-processes its tree on the next run.
+
+### Writing config files
+
+- `-w` / `--write-config` merges the options you invoked into your user config
+  file and then runs normally.
+- `-W` / `--write-dir-config` writes them into a `.picopt.yaml` in each target
+  directory (a file target's parent).
+- `--write-config-file PATH` writes them to an explicit path.
+
+Existing keys and comments in the target file survive the merge. Run-mode flags
+(`-d`, `-L`, verbosity) and the target paths are never persisted.
+
 ## ⌨️ Use Examples
 
 Optimize all JPEG files in a directory:
@@ -288,6 +330,22 @@ timestamps.
 picopt -rStc CBZ,WEBP -x TIFF,CBR,CBZ /Volumes/Media/Comics
 ```
 
+Optimize the same comic library, but cap memory use so multi-GB archives don't
+exhaust RAM:
+
+<!-- eslint-skip -->
+
+```sh
+picopt -rStc CBZ,WEBP -x TIFF,CBR,CBZ --memory-limit 6G /Volumes/Media/Comics
+```
+
+picopt holds each archive plus its decompressed pages in memory while
+optimizing, so large archives spread across many parallel workers can use a lot
+of RAM. `--memory-limit` is an approximate peak-memory target (default:
+two-thirds of detected RAM) that bounds how many large archives run at once — a
+single archive bigger than the whole budget still runs, on its own — and `-j`
+caps the number of parallel workers.
+
 Optimize all files, but only JPEG format files:
 
 <!-- eslint-skip -->

bin/benchmark.py

@@ -0,0 +1,115 @@
+#!/usr/bin/env python3
+"""
+Benchmark picopt wall-clock time and peak RSS over synthetic corpora.
+
+Builds corpora from the test fixtures (an archive-heavy one and an
+image-heavy one), runs picopt as a subprocess, and reports wall time and
+the peak resident set size of the whole process tree.
+
+Usage:
+    uv run python bin/benchmark.py [--archives N] [--images N] [--keep]
+"""
+
+import argparse
+import re
+import shutil
+import subprocess
+import sys
+import time
+from pathlib import Path
+from tempfile import mkdtemp
+
+REPO_ROOT = Path(__file__).parent.parent
+FIXTURES = REPO_ROOT / "tests" / "test_files"
+CONTAINER_SRCS = ("test_cbz.cbz",)
+IMAGE_SRCS = (
+    "test_png.png",
+    "test_jpg.jpg",
+    "test_gif.gif",
+    "test_webp_lossless.webp",
+    "test_animated_gif.gif",
+)
+# /usr/bin/time peak-RSS line: bytes on darwin (-l), kilobytes on linux (-v).
+_IS_DARWIN = sys.platform == "darwin"
+_TIME_ARGS = ("/usr/bin/time", "-l" if _IS_DARWIN else "-v")
+_RSS_RE = re.compile(
+    r"^\s*(\d+)\s+maximum resident set size"
+    if _IS_DARWIN
+    else r"Maximum resident set size[^:]*:\s*(\d+)",
+    re.MULTILINE,
+)
+_RSS_DIVISOR = 1024 * 1024 if _IS_DARWIN else 1024
+
+
+def _build_corpus(root: Path, n_archives: int, n_images: int) -> None:
+    archive_dir = root / "archives"
+    archive_dir.mkdir(parents=True)
+    for i in range(n_archives):
+        src = FIXTURES / "containers" / CONTAINER_SRCS[i % len(CONTAINER_SRCS)]
+        shutil.copy(src, archive_dir / f"comic_{i:04d}.cbz")
+    image_dir = root / "images"
+    image_dir.mkdir(parents=True)
+    for i in range(n_images):
+        src = FIXTURES / "images" / IMAGE_SRCS[i % len(IMAGE_SRCS)]
+        shutil.copy(src, image_dir / f"img_{i:04d}{src.suffix}")
+
+
+def _run(args: tuple[str, ...], target: Path) -> tuple[float, int, int]:
+    """Run picopt over target; return (seconds, peak_rss_mib, returncode)."""
+    start = time.perf_counter()
+    proc = subprocess.run(  # noqa: S603
+        (
+            *_TIME_ARGS,
+            sys.executable,
+            "-c",
+            "from picopt.cli import main; main()",
+            *args,
+            str(target),
+        ),
+        capture_output=True,
+        cwd=REPO_ROOT,
+        check=False,
+    )
+    elapsed = time.perf_counter() - start
+    stderr = proc.stderr.decode(errors="replace")
+    match = _RSS_RE.search(stderr)
+    peak_mib = int(match.group(1)) // _RSS_DIVISOR if match else -1
+    if proc.returncode:
+        print(proc.stdout.decode(errors="replace")[-2000:])
+        print(stderr[-2000:])
+    return elapsed, peak_mib, proc.returncode
+
+
+def main() -> None:
+    """Build corpora and run the benchmark scenarios."""
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--archives", type=int, default=100)
+    parser.add_argument("--images", type=int, default=300)
+    parser.add_argument(
+        "--keep", action="store_true", help="Keep the corpus dir afterwards"
+    )
+    opts = parser.parse_args()
+
+    root = Path(mkdtemp(prefix="picopt-bench-"))
+    _build_corpus(root, opts.archives, opts.images)
+    print(f"corpus: {root} ({opts.archives} archives, {opts.images} images)")
+
+    scenarios = (
+        ("archives cold", ("-rx", "CBZ,PNG,JPEG,GIF,WEBP"), root / "archives"),
+        ("images cold", ("-rx", "PNG,JPEG,GIF,WEBP"), root / "images"),
+        ("images quiet", ("-q", "-rx", "PNG,JPEG,GIF,WEBP"), root / "images"),
+    )
+    try:
+        print(f"{'scenario':<16} {'seconds':>8} {'peak MiB':>9} {'rc':>3}")
+        for name, args, target in scenarios:
+            elapsed, peak, rc = _run(args, target)
+            print(f"{name:<16} {elapsed:>8.2f} {peak:>9} {rc:>3}")
+    finally:
+        if opts.keep:
+            print(f"kept: {root}")
+        else:
+            shutil.rmtree(root, ignore_errors=True)
+
+
+if __name__ == "__main__":
+    main()

bin/roman.py

@@ -116,7 +116,7 @@ def iter_files(path_strs: Sequence[str], spec: PathSpec) -> Generator[Path]:
     for path_str in path_strs:
         path = Path(path_str)
         if not path.exists():
-            print(f"👎  Path does not exist: {path}", file=sys.stderr)  # noqa: T201
+            print(f"👎  Path does not exist: {path}", file=sys.stderr)
             sys.exit(2)
 
         root = path.resolve()
@@ -163,13 +163,13 @@ def _parse_ignore_file(args: Namespace) -> PathSpec:
     if args.ignore_file:
         ignore_path = Path(args.ignore_file)
         if not ignore_path.is_file():
-            print(f"👎  Can't read ignore file: {ignore_path}", file=sys.stderr)  # noqa: T201
+            print(f"👎  Can't read ignore file: {ignore_path}", file=sys.stderr)
             sys.exit(2)
 
     try:
         return build_ignore_spec(ignore_path)
     except OSError as exc:
-        print(f"👎  Failed to parse ignore file: {exc}", file=sys.stderr)  # noqa: T201
+        print(f"👎  Failed to parse ignore file: {exc}", file=sys.stderr)
         sys.exit(2)
 
 
@@ -188,17 +188,17 @@ def main() -> None:
             continue
 
         if not has_description_comment(line2):
-            print(f"🔪  {path}")  # noqa: T201
+            print(f"🔪  {path}")
             offenders.append(path)
 
     if offenders:
-        print(  # noqa: T201
+        print(
             f"\n{len(offenders)} script(s) missing a description comment.",
             file=sys.stderr,
         )
         sys.exit(1)
 
-    print("👍")  # noqa: T201
+    print("👍")
 
 
 if __name__ == "__main__":