Jxl format - #129
Merged
Merged
Conversation
Read and write JPEG XL via the pillow-jxl-plugin codec, as a new auto-discovered plugin. JXL is optimized by default and is now the preferred still conversion target ahead of WEBP and PNG. Only lossless JXL is re-encoded; lossy (XYB encoded) JXL is left alone, the same policy lossy WebP already has. The codec exposes no such signal, so picopt/pillow/jxl.py parses the codestream image header to read xyb_encoded, and refuses files carrying JPEG reconstruction data, whose re-encoding would silently destroy the ability to restore the source JPEG. New --convert-jpeg-to-jxl converts JPEG to JXL by lossless bitstream reconstruction, which is reversible byte for byte. It requires -c JXL as well: picopt otherwise never converts from a lossy source, so -c JXL alone must not sweep up a whole photo library. The codec only takes its reconstruction path when PIL knows the source filename, which it does only for path opens, so the tool opens by path and spills to a temp file for inputs that have none. That makes it work for archive members too. Three smaller things the codec needed: - It registers no save_all handler, and pil_save always passes save_all, which raised KeyError. The single-frame handler is registered for it; animated JXL is unsupported by the codec regardless. - Given no exif kwarg it reads EXIF off the opened image, which quietly defeated --strip-metadata. prepare_info now passes an empty exif. - It rejects palette and other modes, so ImageHandler gained a prepare_image hook, symmetric with prepare_info, to substitute one. Handlers can now declare CONFIG_ENABLED_KEY, a config flag that must be true for them to be probed at all. Failing it leaves the handler out of the probed handler map, which routing already reads as unavailable, so the startup conversion banner stays accurate for free. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
JPEG XL support in browsers and image tooling is still thin, so converting away from WebP, which every browser reads, should be a deliberate act rather than a side effect of asking for -c JXL. JPEG and WEBP are both formats picopt processes by default, which means the usual "name the source with -x" gate cannot express a choice about them: -x JPEG and -x WEBP are no-ops. Each therefore carries its own flag, and a plain -c JXL once again converts only GIF and PNG, as the README has always described. The previous commit had quietly made WEBP a third convert-by-default source. The flag hangs off the handler class, and JxlLossless also serves PNG, GIF and native JXL, which must stay ungated. So the WebP route now points at a JxlFromWebP subclass that exists only to carry CONFIG_ENABLED_KEY. Converting out of JXL into a more widely readable format stays ungated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ajslater
added a commit
that referenced
this pull request
Jul 29, 2026
* bump version
* update deps
* Squashed commit of the following:
commit 0c2b3790c9611f35381d7159c6dea2295e8eb51e
Author: AJ Slater <aj@slater.net>
Date: Thu Jun 26 15:01:52 2025 -0700
internal mozjpeg works, remove external jpeg methods
commit 2a79892c44daf358901c5b1afdac7297e73cbe0e
Merge: 3a01d55 a793e0b
Author: AJ Slater <aj@slater.net>
Date: Thu Jun 26 14:22:38 2025 -0700
Merge branch 'develop' into mozjpeg-lossless-optimization
commit 3a01d55856896d2e4fe94d56cdb60dfd00ffe640
Author: AJ Slater <aj@slater.net>
Date: Wed Jun 11 15:14:40 2025 -0700
use internal mozjpeg. currently broken on mac
* bump news
* remove mozjpeg from build.Dockerfile
* mkdocs
* use dry run lockfile for updated
* configure mkdocs properly
* tilde versions for deps
* add html section to readme
* update deps
* update deps
* revise project urls
* update deps add ty. typecheck with ty
* uv_build
* update scripts, pyproject, eslint, depts, fix lints
* update deps
* update deps ane eslint config
* update deps
* doesn't work with python 3.14
* fix builder dockerfile
* copy debian sources on build properly for ci
* format
* fix tests for new versions of external programs on darwin and somewhat on linux
* fix formatting and use python 3.13 for build
* format
* fix linux test file sizes
* more test size changes for linux
* remove test subdirs
* fix webp sizes in tests for linux
* try to fix pre optimized webp on linux again
* fix sizes again
* one more size
* update deps
* update deps
* add ty to makefile
* delete deprecated pillow code
* support python 3.14, remove old pyright ignores
* bump version and news
* update deps
* update deps
* update deps
* try to get duration info from detect format and PIL
* use webpmux for webp animated
* update deps, update pytest config
* change container test names
* fix PIL jpeg conversion size for mpo test
* replace old test animated webp with one without dropped frames
* enable im2webpanimatedlossless permanently enable webpumx, adjust file sizes so image_dir tests all pass. container tests still not passing
* fix adding info to correct handler clases constructor
* add icecream importer
* bump news and version
* more news
* simplify run_ext_fs for webp no longer needing output_path. compact and simplify creating working paths
* change webp arg order
* use python secure tmpdir for webp temporary files
* fix news version number
* fix test webp sizesfor linux
* update deps
* fix method and variable speling
* type checking fixes
* reduct complexity
* complexipy & dep groups
* reduce complexity of create handler even more
* clean now cleans all caches
* update deps
* fix image format detection
* isolate handler external command tuples for debugging
* fix extracting info from images
* fix conversion to 7zr
* update deps
* remove unused ty ignores
* ty ignores
* update deps
* update deps
* update deps
* update deps
* turn some const sets into tuples for type compatibility
* use tuple instead of list
* type get_config method. remove useless warning raise
* update deps
* fix attrdict type
* update deps
* fix mpo test
* bump version and pin confuse, bump news
* fix test script with test dependency group
* fix lint script with group
* change confuse version specifier
* use dockerfmt. update deps. ruff converts inline ifs to ors
* update build dockerfile
* update deps and add eslint plugins
* add mbake
* remove unused ruff noqas
* Add types with pyre-check
* infer types. bump version and news. add py.typed
* Squashed commit of the following:
commit b7bedc7813be90cb8e922d2dc4fbd3ca55b77b81
Author: AJ Slater <aj@slater.net>
Date: Sat Feb 14 18:26:52 2026 -0800
update devenv
commit 117c504caf8fcd2590e45731b9c4f38c74d5d9bc
Author: AJ Slater <aj@slater.net>
Date: Sat Feb 14 14:45:09 2026 -0800
update devenv scripts
commit 22e7f164469599e98848862589a52f2311ad4e4e
Author: AJ Slater <aj@slater.net>
Date: Sat Feb 14 14:41:27 2026 -0800
add fix merged to update devenv
commit 738ed22be3f3d75c1961339c8921e4d09d7466b8
Author: AJ Slater <aj@slater.net>
Date: Sat Feb 14 14:36:17 2026 -0800
update devenv
commit 848e9911ecc05af9122a1e361755d21925cbcb85
Author: AJ Slater <aj@slater.net>
Date: Sat Feb 14 11:01:58 2026 -0800
remove init script
commit 762550dc406c2345b430833acc961860515c98df
Author: AJ Slater <aj@slater.net>
Date: Sat Feb 14 11:00:08 2026 -0800
update dev scripts
commit 2bc0d768c2c20a30ed1d2f67495002315483f30f
Author: AJ Slater <aj@slater.net>
Date: Sat Feb 14 01:30:49 2026 -0800
use devenv
* update devenv and deps
* update devenv
* update devs and devenv
* update devenv
* add option for ignore archive entry mtimes
* change option name
* update devenv add missing ci script
* fix script hashbang
* move docker-compose to compose remove ignored version
* try running script with simpler sh
* add debug command for circlecir persist
* chown dist before store
* speling
* update devenv
* remove cruft
* update makefile
* update deps
* update devenv
* update devenv
* update devenv
* update devenv
* update devevn
* update devenv
* update devenv & deps
* update deps
* update devevn & deps
* Squashed commit of the following:
commit 5fca5a970e8cf0e1184fd5aae6bd50dd6445a75c
Author: AJ Slater <aj@slater.net>
Date: Wed Apr 8 15:32:09 2026 -0700
disable circleci switch to gha
commit 9f1b3f993f87c8a94eedf8138f14be06def52724
Author: AJ Slater <aj@slater.net>
Date: Wed Apr 8 15:27:18 2026 -0700
skips writing timestamps if no other disk writing operation occurred or needs to
commit 2d40924553cd525696381b04509505eb39e7bed1
Author: AJ Slater <aj@slater.net>
Date: Wed Apr 8 11:46:10 2026 -0700
use env for main picopt test script
* dockerfile becomes builder
* update devenv
* Squashed commit of the following:
commit c82831c17adbf2ca6e44b8fe2734235d8830e6fb
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 23:42:05 2026 -0700
lots of code cleanups mostly match instead of if then chains and Path consistency. One big PathInfo copy constructor that's a little too clever
commit be01ee405c37e19c590f0a667c13775b3b58b6de
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 22:46:28 2026 -0700
minor cleanups for pillow patches
commit a590f6dfea20e800018a900bbf2370f15f3872c5
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 22:42:35 2026 -0700
pdf plugin and tests with variance allowances
commit a6e81ebef927b1c8e4f7c4775895983668f5ebac
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 22:24:01 2026 -0700
add pdf tests
commit 1aa105d2d93791e608119cc95d5555d7d144e866
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 22:05:39 2026 -0700
PDF support
commit 732a14a7629a2cf829d8e2e410cb6d5004900b4d
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 20:50:53 2026 -0700
fix changed calculation for in memory optimization
commit 962fbc9ab9d9df3492abd48304a7f46ea4d778a5
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 20:49:52 2026 -0700
fix top_results adding in walk by filtering none results
commit 74746e6eca906b671bacf8eae6d29037aa4d086e
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 19:27:00 2026 -0700
update trestamps usage
commit 90e6a15f379c8b2d4d9578c4e2e456a04aee8742
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 16:00:16 2026 -0700
minor logging change
commit 484b730535c7b0dd6cdea90e9084a2e0df699fa8
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 15:43:37 2026 -0700
format and colorize doctor
commit 1de8b5c82830d5c4e2c441fdd28497e86d36cce1
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 15:34:22 2026 -0700
doctor mode cli help
commit cb36e7c22b011776d9804be86c707fa5722acebf
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 15:27:31 2026 -0700
version 6 and news
commit 009e61a5d57d49a112f4c26b30092566ccd84f08
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 15:24:40 2026 -0700
break up complexity
commit 97f486c0bd4e5a02df003bdb0f8ff7118a2ef88d
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 14:53:57 2026 -0700
clarify skipped container message
commit 48cd549b807d7aba8c75a333685159145233e078
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 14:35:39 2026 -0700
update deps
commit 3d487c76db3b7ea1ca39d2a21a63c0db3ba8304c
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 14:35:33 2026 -0700
fix mpo test
commit 25681f912016c54ca8dc876597acc54412d5d854
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 14:30:04 2026 -0700
fix MPO being counted as losssless
commit 96306e3c368ffae4822701dccb42ac256bd6f719
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 14:07:43 2026 -0700
don't clean computed config. needed for container in container
commit 4644ae1f48e48b31d42a7153819c52e16efce2b1
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 14:05:45 2026 -0700
move handler cleaning for repack into own function
commit 75ab979f050f10602051633f456a8c8c63a8a348
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 13:56:50 2026 -0700
remove illegal metadata arg from img2webp
commit d7310575400a5188621919660d88dde19b077e32
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 13:30:06 2026 -0700
move convert routes from rar & tar & sevenzip into zip
commit 10d3fd309e09ba9eecb9cb095f0e929feaac0ca4
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 12:51:30 2026 -0700
adjust some png conversion test sizes
commit 4f5377633315e93f9298521ed3b753a8443ee87f
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 12:51:05 2026 -0700
add webp routes for gif and png conversion
commit ba8ddc80d9123dca5ce0efc166e587fa55cc32de
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 12:11:33 2026 -0700
make a StdLibInternalTool for tar and zipfile
commit 6694c2b3f8f9595e79d7ce6fa565a9d49fbb1d2b
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 11:57:00 2026 -0700
consolidate tool base methods, version and path probing
commit 9512e724a226d8b47b3e525de744edc0b8804b33
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 11:24:29 2026 -0700
internal tool version parsing
commit 1bd799ff99fba1296bd31020f6346ede54649575
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 11:21:33 2026 -0700
internal tool version parsing. remove superfluous runtime typechecking code.
commit 9199b039908daac1ed56462baf4ee6a7c49179e2
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 11:20:36 2026 -0700
minor refactor
commit 049add42284d52be56e5beec84f1d170221f20c2
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 10:36:06 2026 -0700
move IS_WEBP_MOERN into CWebpTool from the handler. refactor doctor to be more self contained, use colors more modularly, and report cwebp version modernity inline rather than at the end.
commit cfc68fedb815babee9ac1a095f5f66bbe6a77bd5
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 00:44:56 2026 -0700
fix inheritance checking for import cycle
commit 7f53cc5072875f2c9fc9d329fa0633b1350c1379
Author: AJ Slater <aj@slater.net>
Date: Wed Apr 8 23:58:44 2026 -0700
new doctor and plugin architecture
commit 4f06ef75fccd510aef7b3b768e8c7b876099c66d
Author: AJ Slater <aj@slater.net>
Date: Wed Apr 8 23:57:51 2026 -0700
linted but many tests broken
commit ebb8decc00e1ae0ece2162e172cdc3f1ba078d74
Author: AJ Slater <aj@slater.net>
Date: Wed Apr 8 22:52:28 2026 -0700
format script
* add cancelled if to gha steps
* fix gha file cancelled() function
* install tool for tests
* update devenv
* remove circleci remnamnts
* format dockerfile
* add a set_do_repack() method for the upcoming scheduler to use
* for safety have buffered objects write to the tempfile before replacing the original.
* Scheduler (#103)
* add the scheduler first draft
* Replace walk/dispatch with futures-based scheduler (#101)
Swap multiprocessing.pool.Pool for concurrent.futures.ProcessPoolExecutor
and route all work through a new Scheduler that manages a ContainerNode
tree, three job types (Unpack/OptimizeLeaf/Repack), backpressure, and
per-node failure rollback.
Key changes:
- walk.py: delete _finish_results, _handle_container, _walk_container;
add _enqueue_children callback; walk() creates Scheduler and drains it
- init.py: Pool → ProcessPoolExecutor
- container.py: remove set_task/optimize_contents/_tasks; rename
_hydrate_optimized_path_info → hydrate_optimized_path_info (public API
for scheduler)
- archive.py: always copy unchanged files during walk (archive must be
open); adapt to new hydrate API; drop optimize_contents override
- pdf.py: rename hydrate method to match new public API
- scheduler.py: fix ContainerNode hashability (eq=False); use attribute
access for config; respect handler's own _do_repack flag for animated
image format conversion; delegate hydration to handler subclass
- config: add fail_fast and fail_fast_container options
- Bump treestamps ≥2.5.2 for pickleable GrovestampsConfig
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Add --fail-fast and --fail-fast-container CLI options (#102)
Expose the two new scheduler failure modes on the command line:
- --fail-fast: stop all optimization on the first error
- --fail-fast-container: fail the entire top-level container when
an inner repack fails
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* code cleanups"
* change method ordering for clairity. minor comment edits
* speling
* move old_timestamps into walk as legacy_timestamps
* refactor place format in plugins/base. make hanlder factory it's own class. consolidate report class and walk class.
* consolidate _handler-factory fucntion
* update deps
* code cleanup pass
* warn on digitial signature and encrypted pdf, do not error
* update deps
* use new treestamps methods
* webp tools superclass
* remove unused variable from pdf plugin
* update deps
* fix treestamps use and directory finishing, compacting
* v6.1.0 bump news
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* treestamps 3.0
* bump version and news
* update devenv. convert to bun
* Righttyper typing & corrections (#106)
* bump version and news
* rightyper typechecked
* Fix rightyper-introduced type annotation errors
Fixes 68 basedpyright errors and 51 ty errors introduced by the
automated rightyper pass. Changes are annotation-only.
- PathInfo.__init__ path_info parameter widened from None to PathInfo | None
- Removed spurious self: "picopt.xxx.Y" annotations naming wrong modules
- Simplified **kwargs bogus unions to Any for variadic super().__init__ passthroughs
- config/handlers.py handler_stages/convert_chain/native types widened to proper base types
- scheduler.py _trigger_fail_fast/_cancel_subtree widened Exception to BaseException | None
- detect_format.py filtered image.info keys to str; suppressed dynamic mpinfo attr
- pil_convertible _file_format image_class widened from BmpImageFile to Image
- tar.py removed type[TarGzDetector] cls annotation violating LSP
- tests: PosixPath -> Path; ts:None/config:None -> float|None/dict|None
- Cleaned unused TYPE_CHECKING imports left behind after removing bad annotations
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* correct typing changes
* revert to python 3.10 and remove righttyper
* update devenv
* remove quoted self types
* remove self types
* format & fix
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix bun install on docker and gha compose
* remove package-lock.json
* update devenv
* only copy bun deps first for dockerfile
* update devenv
* update devenv
* Fix BufferedReader pickling error when timestamps file exists (#107)
* fix pickling of ContainerHandler when a timestamps file exists
When a prior picopt run leaves a .picopt_treestamps.yaml, loading it via
ruamel.yaml leaves the YAML instance holding a Reader with an open
BufferedReader. Submitting an UnpackJob (or RepackJob) to the
ProcessPoolExecutor then fails with "cannot pickle 'BufferedReader'
instances" because the handler carries that Grovestamps through its
_timestamps / _skipper attributes.
Add __getstate__ on ContainerHandler that nulls both fields before
pickling. The scheduler owns the real Grovestamps on the main thread;
workers never actually used it (in-archive timestamp consumption
happened in a pickled copy whose mutations never propagated back).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* remove clean_for_repack, superseded by __getstate__
The method nulled the same two fields __getstate__ now nulls at pickle
time, and its one caller always ran on a handler that had already
round-tripped through a worker — so the fields were already None.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* print exception traces
* update deps
* bump news and version
* install poppler utils in docker for pdf tests
* update deps
* format dockerfile
* add bunx support
* bump version and news to v6.2.0
* try with svgo in packages
* Squashed commit of the following:
commit 83a5253785fccc471a6dbd75b4d1eba3074c9e8c
Author: AJ Slater <aj@slater.net>
Date: Wed Apr 29 02:18:19 2026 -0700
bump version and news
commit 9b03aed2141c50e3a088bc6c66bdc097cca7ced4
Author: AJ Slater <aj@slater.net>
Date: Tue Apr 28 20:44:04 2026 -0700
update treestamp loggins
commit 1b7bdc712edc80fc7cf67dc8d7c05f169462d764
Author: AJ Slater <aj@slater.net>
Date: Tue Apr 28 20:32:59 2026 -0700
fix implementation of timestamp loggins to log after not before
commit 8d25937c813ada944d120f3add9ae7b0852d6570
Author: AJ Slater <aj@slater.net>
Date: Tue Apr 28 20:25:28 2026 -0700
Log timestamp load/dump and surface INFO at default verbose=1 (#111)
Treestamps 4 dropped its own load/dump prints, and picopt's new logger
mapped INFO to verbose>=2, so default runs went silent for messages the
old termcolor Printer always showed (config-style force_verbose=True).
- _VERBOSE_LEVEL: bump so verbose=1 (the argparse default) emits INFO,
matching the old printer's "force_verbose" tier.
- walk.py: log "Loading timestamps for: …" before Grovestamps init and
"Dumping timestamps for: …" before dumpf(). INFO renders cyan via the
existing LEVEL_STYLES.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
commit 4415376f5f263cb6dedda5657765c745ca952f6f
Author: AJ Slater <aj@slater.net>
Date: Tue Apr 28 18:41:23 2026 -0700
Pre-walk file count for a determinate progress bar (#110)
* Pre-walk file count for a determinate progress bar
Without a total the bar showed only an indeterminate spinner + count.
Add ``Walk._count_total`` that mirrors ``walk_file``'s recursion gate
(symlinks, timestamp filenames, ignore patterns, recurse flag) so each
non-recursing visit contributes one mark — matching the events the
scheduler dispatches through Reporter — and pass it as ``total=`` to
``make_progress``.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Use os.scandir for the pre-walk count
DirEntry caches ``is_dir`` / ``is_symlink`` from the directory listing,
so deep trees skip an extra ``stat`` per entry — meaningful on slow or
network filesystems.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
commit 6a46d432777367a00ede8b183dfe8ff3bbf6a725
Author: AJ Slater <aj@slater.net>
Date: Tue Apr 28 17:48:59 2026 -0700
Replace termcolor Printer with loguru + rich logger (#109)
The old Printer class wrote dots and messages directly to stdout from
both the main process and worker processes. This change discards it for
a centralized logger modeled on nudebomb's progress branch:
- New picopt/log/ package: shared rich Console, loguru sink, a streaming
CharStreamColumn progress bar, a Stats + render() summary, and a
Reporter that bundles them and dispatches each ReportStats outcome.
- All call sites converted: Printer.{saved,converted,lost,error,skip,
warn,config,...} → logger.* and progress.mark_*. Worker-side dot and
lifecycle calls dropped — workers can't reach the parent's live region,
so per-file progress is now driven from the scheduler when each result
comes back.
- Centralized MARKS table in picopt/log/styles.py drives the streaming
chars, the loguru sink colors, the summary table row colors, and the
--help epilogue legend, so the same outcome reads identically
everywhere. Style choices mirror the old termcolor palette so longtime
users see the same colors for the same outcomes.
- Scheduler now takes a Reporter; Totals removed. report.py is a pure
data class (ReportStats) with no printer dependency.
- doctor.py and the cli help epilogue rewritten with rich (rich.markup
escape() for path strings).
- Grovestamps now constructed with verbose=0 so treestamps's internal
printer doesn't bypass the rich Live region.
- loguru~=0.7 added; termcolor dropped.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
commit 03cdcb018dce4049b6e4de54e71b50df33ddfd15
Author: AJ Slater <aj@slater.net>
Date: Tue Apr 28 17:34:02 2026 -0700
move set_jpeg_xmp into jpeg plugin
commit b28337465aacbad6a65808297dc4bf25115abfd7
Author: AJ Slater <aj@slater.net>
Date: Tue Apr 28 17:06:50 2026 -0700
update devenv. treestamps 4, rich 15
* fix pyright warnings
* ty ignore
* update treestamps
* fix MPO error for 6.3.1
* Migrate to confuse 2.2.0 by replacing AttrDict with PicoptSettings (#114) (#115)
Confuse 2.2.0 made ``AttrDict`` a properly-parameterized generic; an
unparameterized ``AttrDict`` resolves to ``AttrDict[str, object]``, so
every downstream access — ``cfg.bigger``, ``int(cfg.jobs)``,
``cfg.paths`` flowing into ``Iterable[str]``, etc. — now reads as
``object`` and stops type-checking. We had ~31 basedpyright errors after
the bump, all rooted at AttrDict consumers.
Fix: keep using confuse for what it's good at — YAML / env / CLI parsing,
type coercion, requiredness — and convert the validated ``AttrDict``
into a typed frozen dataclass once. Every downstream module then takes
``PicoptSettings`` instead of ``AttrDict``.
- ``picopt/config/settings.py``: new ``PicoptSettings`` (and nested
``ComputedSettings`` / ``IgnorePatterns``) frozen dataclasses mirroring
the existing ``MappingTemplate`` schema. ``Sequence(...)`` fields are
``tuple[X, ...]`` because the dataclass is frozen. ``handler_stages``
is typed ``dict[Any, Any]`` to avoid an import cycle with
``picopt.plugins.base``; consumers know its concrete shape.
- ``picopt/config/__init__.py``: ``PicoptConfig.get_config`` now returns
``PicoptSettings``. New ``_settings_from_attrdict`` does the
field-by-field copy (typed ``Any`` so it can read AttrDict attributes
directly — the conversion *is* the schema escape hatch).
- All consumers (``picopt/path.py``, ``picopt/report.py``, ``picopt/walk/*``,
``picopt/plugins/base/{handler,container}.py``) replace
``from confuse(.templates) import AttrDict`` with
``from picopt.config.settings import PicoptSettings``.
- ``Walk._init_timestamps`` builds a shallow ``Mapping[str, Any]`` for
Grovestamps' ``program_config=`` so we don't have to serialize the
whole frozen dataclass (especially the computed sub-fields, which
carry ``re.Pattern`` and class-keyed ``dict`` objects).
- ``is_path_ignored`` switched from ``ignore and bool(...)`` to
``ignore is not None and bool(...)`` — its return type is now
honestly ``bool``, not ``bool | re.Pattern | None``.
- ``Walk._dump_timestamps`` joins the dumpf return paths via ``str(p)``
(treestamps' ``dumpf`` returns ``tuple[Path, ...]``).
pyproject pin: ``confuse~=2.1.0,<2.2.0`` → ``confuse~=2.2.0``.
Verification: ruff clean, basedpyright clean (0 errors, was 31),
``uv run pytest --deselect …test_timestamp_parents`` 149 passed
(deselected test is a pre-existing treestamps v4 regression),
``radon cc`` all A/B, CLI smoke run with env var + CLI flag still
layers correctly.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Drop dead isinstance check; split Scheduler._submit_ready_job (#116)
Two warnings flagged by `make typecheck complexity`:
- ``picopt/config/__init__.py``: confuse 2.2.0 already types
``config.get(MappingTemplate)`` as ``AttrDict[str, object]``, so the
defensive ``isinstance(ad, AttrDict)`` check after it was statically
always-true (reportUnnecessaryIsInstance). Drop it and the now-unused
``AttrDict`` import.
- ``picopt/walk/scheduler.py``: ``_submit_ready_job`` was rank C in
radon. Split into three:
* ``_drop_cancelled_ready_job`` — counter cleanup for a leaf of a
cancelled subtree (UnpackJob/RepackJob skip; only leaf jobs
decrement the parent counter).
* ``_track_submitted_job`` — record an in-flight future under the
right map and update node state.
* ``_submit_ready_job`` — orchestrate: pop, cancelled-skip, submit,
track.
``_submit_ready_job`` is now rank A; the new helpers are A/B; behavior
unchanged.
Verified: ``make typecheck`` 0/0/0; ``make complexity`` no findings;
``uv run ruff check picopt/`` clean; ``uv run pytest`` 150 passed,
6 skipped.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* update deps
* bump version and news to 6.4.0
* update devenv
* Format argparse help with rich-argparse (#118)
Switch picopt's CLI to RawDescriptionRichHelpFormatter so the help
output is colorized. A PicoptHelpFormatter subclass adds one extra
highlight regex matching every registered format string (PNG, ZIP,
WEBP, …) under the `metavar` named group, so format names mentioned
inside help text share the color of their corresponding FORMATS /
EXTRA_FORMATS / CONVERT_TO metavars.
Rewrite the dot-color-key / doctor-mode epilog as Rich markup instead
of capturing ANSI from a Console. rich-argparse renders descriptions
and epilogs through `console.use_theme(Theme(self.styles))`, so the
epilog can use [argparse.groups], [argparse.prog], and [argparse.args]
to keep its section header, program name, and `doctor` subcommand
visually consistent with how the rest of the help is rendered.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* bump version and news to 6.5.0
* typecheck
* Defer animated WebP duration extraction to handler (#119)
detect_format eagerly iterated every frame of every animated image to
populate info["durations"], which only WebPMuxAnimatedLossless ever
consumed -- and that handler already had a webpmux-based fallback
(_read_durations) for when PIL's frame iteration was unreliable.
Removing the eager extraction speeds up format detection on animated
images (animated PNG ~11x, animated GIF ~4x in local microbenchmarks)
and simplifies the webp.walk path to always use the webpmux subprocess
fallback, which is the more reliable source per the original "PIL
frequently fails to populate per-frame durations on WebP" comment.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* bump news for animated deferall
* Cache file header bytes on PathInfo for detectors (#120)
PdfDetector was opening every file and reading 1024 bytes for every
detection attempt -- including non-PDF files where it would read,
fail the magic check, and return None. On a directory of non-image
non-archive files, this was ~14us per file (88% of the non-PIL
detector chain cost).
PathInfo now lazily reads up to 4 KB of file head into _header_bytes
on first access. PdfDetector consumes path_info.header_bytes() instead
of reopening the file.
Local benchmark of running the full non-PIL detector chain on
non-image files (pyproject.toml, README.md, uv.lock, Makefile):
~16us before -> ~2us after (8x).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* bump news
* Single Image.open in _extract_image_info; read metadata before verify (#121)
PIL's verify() closes its internal fp for some Path-opened formats
(notably GIF), which then breaks lazy attrs like is_animated and
n_frames. The previous code worked around this by opening the file
twice -- once to verify, again to read format/info.
Reading the lazy attrs before verify() preserves the safety check
(verify still runs and propagates corruption errors via the same
exception path) while collapsing to a single Image.open. Confirmed
on Pillow 12.2 against GIF (still + animated), TIFF (single + multi-
frame), MPO, animated PNG, animated WebP, JPEG, BMP, PPM.
_extract_image_info per-file (warm cache, microbenchmark):
test_animated_gif.gif 165us -> 95us
test_animated_png.png 145us -> 113us
test_animated_webp.webp 66us -> 30us
test_png.png 74us -> 35us
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* bump news
* update devenv and fix typechecking
* remove use of prettier sh plugin'
gi tpush
* fix data loss when converting without external tools
pil_save short-circuited format conversion when the input format was in
the handler's accepted set, even when the target format differed. This
caused PNG→WebP via PIL fallback to return the original PNG buffer
unchanged, which the cleanup code then deleted without writing a WebP.
Also fix pipeline selection ignoring tool.required=False, which caused
PNG optimization to be skipped entirely when pngout was absent despite
pyoxipng being available. Add PIL fallback for GIF when gifsicle is
missing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* doctor: show platform-specific install hints for missing tools
Detect the package manager (brew/apt/dnf) and display the install
command next to each missing external tool in `picopt doctor` output.
Also remove obsolete mozjpeg/jpegtran system install instructions from
the README — mozjpeg is now a bundled Python dependency that installs
automatically with picopt.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* decomplexify doctor
* update deps
* bump version and news
* update devenv. remove old prettier sh plugin
* replace pyoxipng with oxipng-pybind (#124)
Swaps the deprecated pyoxipng binding for oxipng-pybind, which wraps
oxipng v10.1.1 (vs v9.x) and is actively maintained. The module import
name is unchanged (import oxipng), so the only code change needed was
fixing one deprecated call: StripChunks.safe() → StripChunks.safe (enum
value, no parentheses). All 150 tests pass.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* update deps
* bump news and version to v6.5.2
* update deps
* update deps
* Add memory-aware scheduling to prevent OOM on large archives
Optimizing large archives (e.g. multi-GB comic books) could exhaust RAM
and get the process OOM-killed. Each container holds its whole archive,
all decompressed members, the optimized copies, and the output buffer in
memory at once, replicated into each parallel worker, and nothing bounded
the in-flight total beyond a future-count cap.
Give the scheduler a memory budget: estimate each top-level archive's peak
resident memory (~3x its on-disk size) and only admit a new one while it
fits the budget, with a forward-progress guarantee so a single archive
larger than the whole budget still runs (alone). The gate scans past
budget-blocked items so in-flight work still frees the budget, avoiding
both deadlock and head-of-line blocking.
Add a --memory-limit option (e.g. 8G, 512M) that sets the budget as an
approximate peak-memory target; the default is two-thirds of detected RAM.
It is excluded from the timestamps config so it never invalidates stamps.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* update deps and format
* Fix basedpyright and ty diagnostics
Convert mypy-style `# type: ignore[arg-type]` comments in the scheduler
memory test to the project's paired `# pyright: ignore # ty: ignore`
convention (neither checker honors mypy-style comments), and make the
`_charge_info` match exhaustive with an explicit `case _` for the
intentionally-exempt RepackJob path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* update devenv
* remove pytest-gitignore
* Fix critical data-loss bugs in file replacement and archive repacking
- Write optimized results to a sibling temp file and atomically replace
the original; a crash, OOM-kill, or full disk mid-write can no longer
destroy the user's file.
- Keep members whose optimization errored (and nested containers whose
repack failed) in the repacked archive instead of silently dropping them.
- Open nested archives from their in-memory buffer; archives inside
archives previously always failed to unpack with FileNotFoundError.
- Discarded conversion results (bigger output, dry-run) no longer stat a
never-written path and are reported as skipped instead of errors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Reduce _select_pipeline_for_handler cognitive complexity
Extract per-tier tool picking into _pick_tier_tool and drop the
redundant empty-pipeline guard; behavior is unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Fix scheduler and walk error-path correctness
- A member that fails format detection (e.g. a decompression bomb) is
recorded as one error and passed through to the repacked archive
instead of aborting the entire run and losing all timestamps.
- Directories and containers with errored children no longer get
timestamps; previously the compacted dir stamp covered failed files
so they were silently skipped on every subsequent run.
- Failed top-level containers now notify their directory tracker, which
previously leaked and blocked ancestor timestamp finalization.
- Nested container repacks hydrate through the parent handler so
conversion renames (e.g. inner CBR -> CBZ) are no longer lost.
- Requested container conversions repack even when every member was
already optimized; CBR -> CBZ previously did nothing in that case.
- Ignore patterns now respect filesystem case sensitivity (the flag
was inverted), timestamps files are matched by basename, and FIFOs/
sockets are skipped instead of blocking the walk forever.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Preserve archive metadata fidelity through repack
- Archive comments (zip, rar) are read during walk and written back on
repack; _set_comment previously had no call site so comments were
always destroyed (CBZ comments commonly hold ComicBookInfo tags).
- Repack preserves original member order via a walk-time archive_index
on PathInfo; the optimized-contents set previously scrambled order,
which broke EPUB's mimetype-first requirement and reproducibility.
- Member times before 1980 (or missing) are clamped to the zip epoch
instead of crashing zipfile on conversion.
- Converted members choose compression by content: text deflates,
already-compressed image formats are stored. Previously rar/tar/7z
conversions wrote every member uncompressed, and --strip-metadata
wastefully deflated incompressible images.
- Legacy zip member names that are really UTF-8 misread as CP437 are
repaired at repack instead of being re-encoded as mojibake forever.
- Tar links survive repack: symlinks/hardlinks keep their type, target,
and zero size; link targets are no longer duplicated into link
entries (or crashes on external targets).
- 7z member mtimes are restored after repack; py7zr's writef API stamps
members with now() and offers no public override.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Fix config layering, validation, and doctor accounting
- Config-file and env values for boolean options and verbose are no
longer silently shadowed: un-passed CLI flags now default to None,
which confuse drops, so lower layers stay visible. CLI flags that ARE
passed still override everything.
- -I/--no-default-ignores without -i no longer aborts at startup (the
computed ignore patterns are validly None).
- Unparseable --memory-limit values abort with a clean config error
instead of silently falling back to auto, and unparseable --after
values no longer dump a raw dateutil traceback. Timezone-aware
--after values are interpreted correctly instead of as local time.
- picopt doctor counts requirements per pipeline tier instead of per
tool; tools within a tier are alternatives, so healthy installs no
longer report missing-required tools and exit 1.
- Lowercase format lists are accepted for -x and -c exactly as for -f.
- README no longer documents the removed PICOPT_ENABLE_IMG2WEBP
environment variable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Fix webp lossless detection, png depth parsing, and handler edge cases
- is_lossless() parses the RIFF chunk structure instead of searching
for byte strings. The old code searched the whole file via mmap for
filesystem input (metadata bytes could fake losslessness) but only
128 bytes for in-memory input (a large ICC/EXIF chunk before VP8L
misdetected lossless in-archive WebPs as lossy). It also closed the
caller's buffer and crashed mmap on empty files.
- png_bit_depth() returns None for truncated PNGs instead of a fake
0 depth.
- run_ext_fs() cleans up spilled temp files when the external tool
fails instead of leaking them.
- --preserve survives chown permission failures (non-root users) and
still restores mode and mtime instead of reporting the whole
optimization as an error.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Make PDF and SVG optimization safer
- Repacking preserves PDF encryption: owner-password-restricted PDFs
open without a password and were silently saved decrypted. qpdf
cannot decode streams while preserving encryption, so encrypted PDFs
get the weaker (but protected) rewrite.
- Optimized JPEGs are written back with their original /DecodeParms
(e.g. /ColorTransform), which pikepdf's write() otherwise drops.
- Signature detection also looks for /Type /Sig objects, catching
signed PDFs from producers that never set the AcroForm SigFlags bit.
- svgo runs with an explicit config instead of the default preset:
removeViewBox is disabled (it breaks responsive scaling), and title/
desc/metadata elements follow the --strip-metadata flag instead of
the preset's whims.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Survive symlink loops and unreadable directories
- walk_dir, the progress pre-count, and the legacy timestamp import
track visited directories by (st_dev, st_ino); symlink cycles and
duplicate links are walked once instead of recursing until the
filesystem errors (or, in the pre-count, forever).
- The legacy timestamp import skips unreadable paths with a warning
instead of crashing the whole run at startup, and matches ignore
patterns with the same case-sensitivity fix as the walk skipper.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Revive in-archive entry skipping and timestamp consumption
ContainerHandler.walk() always executes in a worker process, but
__getstate__ dropped the Grovestamps (unpicklable ruamel state) it
needs, so member entry-mtime skipping, in-archive .picopt_treestamps
consumption, and the timestamps_ignore_archive_entry_mtimes option
were all dead code.
Archive handlers now carry an ArchiveStamps slice instead: the tree's
picklable config plus its dump_dict() serialization (the same
config-headed mapping treestamps round-trips through disk). A real,
detached Treestamps is rebuilt lazily in the worker — never set() or
dumped, so it never touches the WAL — preserving full treestamps
semantics: ancestor-max lookups and config-checked loading of stamp
files found inside archives.
Members whose entry mtimes predate the archive's stamp are now
noop-copied instead of re-optimized when a stamped archive is
modified, stamp files inside archives are consumed and dropped on
repack, and -E disables entry-mtime skipping as documented.
timestamps_ignore_archive_entry_mtimes joins TIMESTAMPS_CONFIG_KEYS
since flipping it changes which members get optimized.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Improve scheduler-thread and archive throughput
- Archive member format detection (PIL open + verify per member) moves
into the unpack workers via predetect_format(), which caches onto
PathInfo; the scheduler thread previously decoded every member of
every archive serially.
- The progress pre-count no longer re-walks the whole tree when the
progress bar is disabled (-q).
- Top-level results no longer pickle their optimized bytes back to the
main process just to be discarded; only in-container entries return
data for their parent's repack.
- Items deferred by the memory gate wait in a FIFO gated queue retried
head-first per tick instead of rescanning the entire ready queue on
every completion (O(queue) per event under memory pressure).
- 7z archives extract all members in one pass; solid archives
previously decompressed from the start for every single-target
extract, O(n^2) over members. The extraction cache is dropped before
the handler pickles back to the main process.
- Add bin/benchmark.py: wall-clock + per-run peak RSS harness over
synthetic fixture corpora, for measuring future changes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Extract dir timestamping, wire staging cleanup, untangle reporter dispatch
- The per-directory completion tracking and compacted timestamp writes
move from the scheduler into walk/dir_timestamps.DirTimestamper; the
scheduler delegates its begin/seal/cancel API.
- All five container pending-counter decrements route through one
_child_done() method; _maybe_start_repack's state guards make it safe
on every completion path.
- The staging-dir cleanup design is finally wired: handlers expose
get_staging_dir(), the scheduler stores it on the node after unpack,
and rollback/fail-fast/final cleanup actually remove worker tmp dirs.
Animated-WebP frame directories previously leaked on cancellation.
- Reporter outcome dispatch is an explicit match instead of
getattr(f"record_{kind}") string dispatch that failed silently on
renames.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Split the webp plugin into a package
plugins/webp.py (759 lines, five handlers, four pack strategies) becomes
plugins/webp/: const, tools, static, and animated modules with the
PLUGIN descriptor in __init__. Plugin discovery now accepts subpackages.
The CWebPTool.probe() -> IS_MODERN_CWEBP class-attribute side effect is
replaced by an is_modern flag on the shared, probed-once CWEBP_TOOL
singleton that WebPLossless's PIPELINE holds and both consuming handlers
read at construction time. The tools no longer import handler classes at
all — they duck-type on the args-provider methods — removing the
tool/handler import cycle the isinstance checks required.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Deduplicate handler routing and document the PathInfo contract
- plugins.pick_route_handler is the one routing decision: the factory
uses it at runtime and config/handlers uses it to build the startup
formats summary, replacing an 8-parameter re-derivation whose logic
had already drifted from the factory's (it ignored empty-pipeline
availability). The log can no longer disagree with the walk.
- PathInfo's class docstring documents the mutable-identity contract:
instances cross the process boundary and are mutated by the scheduler
after hydration, with cache invalidation owned by rename()/set_data().
- computed.handler_stages is structurally typed dict[type, tuple];
the concrete Handler/Tool annotation is impossible because
plugins.base imports settings and basedpyright's reportImportCycles
counts even TYPE_CHECKING edges.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Memoize tool probes
probe() caches its ToolStatus per instance and concrete probes move to
_probe(). Tools are module singletons, so each external tool spawns its
--version subprocess once per process — a prerequisite for per-directory
config files, whose confuse rebuilds rerun the pipeline probe loop.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Add per-directory .picopt.yaml config files and config-writing flags
Ported from nudebomb's implementation (same confuse + treestamps stack).
A .picopt.yaml placed in a target directory applies to that directory
and everything beneath it. Files are discovered from each processed
file's directory up to the CLI target root (never above it); deeper
directories win over shallower ones, and env vars and CLI options still
win over every directory file — implemented by rebuilding a fresh
confuse Configuration per unique directory chain through the extracted
_build_config layering core. DirConfig caches resolutions per directory
with an identity fast path when no config applies, and isolates
malformed files (logged once, recorded in stats, fall back to run-wide
settings). Run-mode keys (dry_run, list_only, timestamps) are pinned to
run-level values: a per-directory dry run with run-level timestamps
would stamp files that were never optimized.
Per-directory settings reach everything: guards (ignore/symlinks/
recurse) via per-directory skippers and the pre-count mirror, handlers
via the factory's settings parameter (formats/convert_to/bigger/
keep_metadata/... travel into workers), archive members by inheriting
their container's config (nesting inherits automatically), and repacks
run under the handler's own config. .picopt.yaml files are skipped as
optimization targets by basename.
With -t, a sha256 fingerprint of every .picopt.yaml under the targets
joins the treestamps program-config check, so editing, adding, or
removing a directory config re-processes its tree (over-invalidation,
never a wrong-skip).
New write flags, all validate-before-persist with round-trip YAML
merges that preserve comments and unrelated keys, written owner-only:
-w/--write-config (user config), -W/--write-dir-config (a .picopt.yaml
in each target directory), and --write-config-file PATH. Run-mode
flags and target paths are never persisted.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Add fail-fast, cancellation, idempotency, and pixel-identity tests
- --fail-fast cancels every live top-level container and clears both
scheduler queues on the first repack error; --fail-fast-container
escalates an inner failure to its subtree root while leaving sibling
top-level containers untouched; a leaf result arriving after its
parent's cancellation is dropped without touching the parent's
contents.
- Optimization converges: a second pass may only shrink files (the
multi-tool png pipeline legitimately shaves another byte when one
tool's output enables another) and every file reaches its fixed
point by the third pass.
- Lossless paths are pixel-exact: png optimization, png-to-webp, and
gif-to-png conversions decode to identical RGBA pixels.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Update NEWS for v6.6.0
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Add missing v6.6.0 news items
Fill in five user-facing fixes that landed since v6.5.2 but were absent
from the v6.6.0 entry: archive conversions recompress members by content,
--preserve survives non-root chown failures, temp files and animated-WebP
frame dirs are cleaned up on failure/cancellation, timezone-aware --after
is honored, and -I without -i no longer aborts at startup.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* version 6.6.0
* Fix repeated "Optimizing formats" banner with -W and planted .picopt.yaml
The verbose "Optimizing formats" banner is a run-wide summary, but it was
emitted from set_format_handler_map(), which _build_config() re-runs for
every per-directory .picopt.yaml resolution during the walk. With -W (or
any planted directory config in the tree root), every directory finds a
config in its ancestry, skips the "reuse run-wide settings" fast path, and
rebuilds the config — reprinting the banner once per directory.
Gate the banner behind a print_summary flag so only the top-level
get_config() build emits it. Move the summary computation into
_log_formats_summary() so the per-directory rebuilds skip it entirely.
Add a regression test asserting the banner prints exactly once under -W.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* bump version to 6.6.1
* bump news
* Fix animated WebP optimization failing on zero-duration frames
img2webp rejects any frame duration <= 0 ("Invalid negative duration"),
aborting the whole animated-WebP repack. Some sources (e.g. scraped
animated WebPs) carry 0ms frame durations, so clamp each duration up to
img2webp's 1ms minimum in Img2WebPAnimatedLossless.img2webp_args.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* update deps
* Clear basedpyright typecheck warnings
- Use a real PicoptSettings (not SimpleNamespace) in the img2webp
duration test so config assignment type-checks.
- Mark the module-level pytest autouse fixtures with
`# pyright: ignore[reportUnusedFunction]`; they are used by pytest,
not referenced by name.
- Disable reportImplicitStringConcatenation: ruff format canonicalizes
adjacent string literals to implicit concatenation, so the check can
never be satisfied.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix news formattign
* Reduce cyclomatic complexity of four rank-C functions
radon cc flagged four functions at rank C. Extract behavior-preserving
helpers to bring each to rank A/B:
- WalkSkipper._is_skippable (16): split into _skip_reason and the
disk-only _skip_reason_fs.
- Walk._count (12): extract the recursion-stop predicate _count_stops_here.
- pick_route_handler (12): fold the convert gate into _pick_convert_handler
and extract _can_pack_repack for the repack check.
- Handler._cleanup_after_optimize (11): extract the _should_replace predicate.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Print verbose config summaries only once, not per directory config
Per-directory .picopt.yaml resolution rebuilds the confuse config for
every directory in the tree, repeating the timestamps, ignore, after,
and memory-limit log lines each time. Gate them on the existing
print_summary flag, as set_format_handler_map already does.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* update deps and bump version to 6.6.2
* Add v6.6.2 news entry for repeated verbose summaries fix
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix timestamps silently invalidated by tree-root .picopt.yaml configs
The treestamps program config was recorded from the run-level CLI
config, so identical options expressed via a tree root's .picopt.yaml
instead of flags always mismatched and every stamp was silently
discarded, re-optimizing whole trees.
Each tree now records its own effective config: the new Grove builds
one Treestamps per top path from DirConfig.get_tree_settings, which
resolves the root's .picopt.yaml beneath CLI and env. timestamps: true
at a tree root now enables stamping without -t (dry run and list still
disable it, and subdirectory configs still can't toggle it mid-walk).
The directory-config fingerprint is now per tree and value-based: it
hashes the parsed picopt section of each config strictly below the
root, so comment and formatting edits, -W rewrites, running with a
different set of top paths, and file-vs-directory targets of the same
tree no longer invalidate stamps — only real option changes do. The
root's own config is excluded because its options are recorded as
values.
Stamps discarded on a config mismatch now warn with the differing keys
(treestamps 4.1.1); stdlib logging is intercepted into loguru so the
warning renders through the shared console, and log messages escape
Rich markup so bracketed names like "Title [en]" display intact.
Migration: existing stamp files mismatch once after this upgrade (now
loudly); a single -N run preserves old stamps instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T7R5s6frL7jEykysAzjttq
* update deps
* Report corrupt images as clean warnings instead of dumping tracebacks
PIL raises a plain OSError when it recognizes an image format but the
contents are corrupt (e.g. a BMP with pixel depth 0), which escaped the
detection probe and printed a full stack trace. Detection now wraps it
in UnreadableImageError, reported as a one-line warning that is also
counted and listed in the summary; the file is skipped untouched.
The per-file exception guards now skip the stack trace for any
PicoptError — extending the contract cli.main already had: expected
conditions print their message, only unexpected exceptions dump stacks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T7R5s6frL7jEykysAzjttq
* version 6.6.3
* update deps
* Fix ty and remark-lint warnings
- Drop the `cls: type[Any]` annotation on the RarDetector/ZipDetector
`identify` classmethods; ty flagged the widened first parameter as an
invalid override of `Detector.identify`.
- Replace remark's `no-duplicate-headings` with
`no-duplicate-headings-in-section`. A changelog necessarily repeats
`### Fixes` under each version; the in-section rule still catches real
duplicates.
Also includes the in-progress dependency bumps and ruff rule additions that
were already in the working tree.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Jxl format (#129)
* Add JPEG XL support
Read and write JPEG XL via the pillow-jxl-plugin codec, as a new
auto-discovered plugin. JXL is optimized by default and is now the
preferred still conversion target ahead of WEBP and PNG.
Only lossless JXL is re-encoded; lossy (XYB encoded) JXL is left alone,
the same policy lossy WebP already has. The codec exposes no such signal,
so picopt/pillow/jxl.py parses the codestream image header to read
xyb_encoded, and refuses files carrying JPEG reconstruction data, whose
re-encoding would silently destroy the ability to restore the source JPEG.
New --convert-jpeg-to-jxl converts JPEG to JXL by lossless bitstream
reconstruction, which is reversible byte for byte. It requires -c JXL as
well: picopt otherwise never converts from a lossy source, so -c JXL alone
must not sweep up a whole photo library. The codec only takes its
reconstruction path when PIL knows the source filename, which it does only
for path opens, so the tool opens by path and spills to a temp file for
inputs that have none. That makes it work for archive members too.
Three smaller things the codec needed:
- It registers no save_all handler, and pil_save always passes save_all,
which raised KeyError. The single-frame handler is registered for it;
animated JXL is unsupported by the codec regardless.
- Given no exif kwarg it reads EXIF off the opened image, which quietly
defeated --strip-metadata. prepare_info now passes an empty exif.
- It rejects palette and other modes, so ImageHandler gained a
prepare_image hook, symmetric with prepare_info, to substitute one.
Handlers can now declare CONFIG_ENABLED_KEY, a config flag that must be
true for them to be probed at all. Failing it leaves the handler out of
the probed handler map, which routing already reads as unavailable, so the
startup conversion banner stays accurate for free.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Gate WebP to JXL conversion behind --convert-webp-to-jxl
JPEG XL support in browsers and image tooling is still thin, so converting
away from WebP, which every browser reads, should be a deliberate act
rather than a side effect of asking for -c JXL.
JPEG and WEBP are both formats picopt processes by default, which means
the usual "name the source with -x" gate cannot express a choice about
them: -x JPEG and -x WEBP are no-ops. Each therefore carries its own flag,
and a plain -c JXL once again converts only GIF and PNG, as the README has
always described. The previous commit had quietly made WEBP a third
convert-by-default source.
The flag hangs off the handler class, and JxlLossless also serves PNG, GIF
and native JXL, which must stay ungated. So the WebP route now points at a
JxlFromWebP subclass that exists only to carry CONFIG_ENABLED_KEY.
Converting out of JXL into a more widely readable format stays ungated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* update deps
* update deps
* update deps
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* update deps
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
ajslater
added a commit
that referenced
this pull request
Aug 3, 2026
* remove mozjpeg from build.Dockerfile
* mkdocs
* use dry run lockfile for updated
* configure mkdocs properly
* tilde versions for deps
* add html section to readme
* update deps
* update deps
* revise project urls
* update deps add ty. typecheck with ty
* uv_build
* update scripts, pyproject, eslint, depts, fix lints
* update deps
* update deps ane eslint config
* update deps
* doesn't work with python 3.14
* fix builder dockerfile
* copy debian sources on build properly for ci
* format
* fix tests for new versions of external programs on darwin and somewhat on linux
* fix formatting and use python 3.13 for build
* format
* fix linux test file sizes
* more test size changes for linux
* remove test subdirs
* fix webp sizes in tests for linux
* try to fix pre optimized webp on linux again
* fix sizes again
* one more size
* update deps
* update deps
* add ty to makefile
* delete deprecated pillow code
* support python 3.14, remove old pyright ignores
* bump version and news
* update deps
* update deps
* update deps
* try to get duration info from detect format and PIL
* use webpmux for webp animated
* update deps, update pytest config
* change container test names
* fix PIL jpeg conversion size for mpo test
* replace old test animated webp with one without dropped frames
* enable im2webpanimatedlossless permanently enable webpumx, adjust file sizes so image_dir tests all pass. container tests still not passing
* fix adding info to correct handler clases constructor
* add icecream importer
* bump news and version
* more news
* simplify run_ext_fs for webp no longer needing output_path. compact and simplify creating working paths
* change webp arg order
* use python secure tmpdir for webp temporary files
* fix news version number
* fix test webp sizesfor linux
* update deps
* fix method and variable speling
* type checking fixes
* reduct complexity
* complexipy & dep groups
* reduce complexity of create handler even more
* clean now cleans all caches
* update deps
* fix image format detection
* isolate handler external command tuples for debugging
* fix extracting info from images
* fix conversion to 7zr
* update deps
* remove unused ty ignores
* ty ignores
* update deps
* update deps
* update deps
* update deps
* turn some const sets into tuples for type compatibility
* use tuple instead of list
* type get_config method. remove useless warning raise
* update deps
* fix attrdict type
* update deps
* fix mpo test
* bump version and pin confuse, bump news
* fix test script with test dependency group
* fix lint script with group
* change confuse version specifier
* use dockerfmt. update deps. ruff converts inline ifs to ors
* update build dockerfile
* update deps and add eslint plugins
* add mbake
* remove unused ruff noqas
* Add types with pyre-check
* infer types. bump version and news. add py.typed
* Squashed commit of the following:
commit b7bedc7813be90cb8e922d2dc4fbd3ca55b77b81
Author: AJ Slater <aj@slater.net>
Date: Sat Feb 14 18:26:52 2026 -0800
update devenv
commit 117c504caf8fcd2590e45731b9c4f38c74d5d9bc
Author: AJ Slater <aj@slater.net>
Date: Sat Feb 14 14:45:09 2026 -0800
update devenv scripts
commit 22e7f164469599e98848862589a52f2311ad4e4e
Author: AJ Slater <aj@slater.net>
Date: Sat Feb 14 14:41:27 2026 -0800
add fix merged to update devenv
commit 738ed22be3f3d75c1961339c8921e4d09d7466b8
Author: AJ Slater <aj@slater.net>
Date: Sat Feb 14 14:36:17 2026 -0800
update devenv
commit 848e9911ecc05af9122a1e361755d21925cbcb85
Author: AJ Slater <aj@slater.net>
Date: Sat Feb 14 11:01:58 2026 -0800
remove init script
commit 762550dc406c2345b430833acc961860515c98df
Author: AJ Slater <aj@slater.net>
Date: Sat Feb 14 11:00:08 2026 -0800
update dev scripts
commit 2bc0d768c2c20a30ed1d2f67495002315483f30f
Author: AJ Slater <aj@slater.net>
Date: Sat Feb 14 01:30:49 2026 -0800
use devenv
* update devenv and deps
* update devenv
* update devs and devenv
* update devenv
* add option for ignore archive entry mtimes
* change option name
* update devenv add missing ci script
* fix script hashbang
* move docker-compose to compose remove ignored version
* try running script with simpler sh
* add debug command for circlecir persist
* chown dist before store
* speling
* update devenv
* remove cruft
* update makefile
* update deps
* update devenv
* update devenv
* update devenv
* update devenv
* update devevn
* update devenv
* update devenv & deps
* update deps
* update devevn & deps
* Squashed commit of the following:
commit 5fca5a970e8cf0e1184fd5aae6bd50dd6445a75c
Author: AJ Slater <aj@slater.net>
Date: Wed Apr 8 15:32:09 2026 -0700
disable circleci switch to gha
commit 9f1b3f993f87c8a94eedf8138f14be06def52724
Author: AJ Slater <aj@slater.net>
Date: Wed Apr 8 15:27:18 2026 -0700
skips writing timestamps if no other disk writing operation occurred or needs to
commit 2d40924553cd525696381b04509505eb39e7bed1
Author: AJ Slater <aj@slater.net>
Date: Wed Apr 8 11:46:10 2026 -0700
use env for main picopt test script
* dockerfile becomes builder
* update devenv
* Squashed commit of the following:
commit c82831c17adbf2ca6e44b8fe2734235d8830e6fb
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 23:42:05 2026 -0700
lots of code cleanups mostly match instead of if then chains and Path consistency. One big PathInfo copy constructor that's a little too clever
commit be01ee405c37e19c590f0a667c13775b3b58b6de
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 22:46:28 2026 -0700
minor cleanups for pillow patches
commit a590f6dfea20e800018a900bbf2370f15f3872c5
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 22:42:35 2026 -0700
pdf plugin and tests with variance allowances
commit a6e81ebef927b1c8e4f7c4775895983668f5ebac
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 22:24:01 2026 -0700
add pdf tests
commit 1aa105d2d93791e608119cc95d5555d7d144e866
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 22:05:39 2026 -0700
PDF support
commit 732a14a7629a2cf829d8e2e410cb6d5004900b4d
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 20:50:53 2026 -0700
fix changed calculation for in memory optimization
commit 962fbc9ab9d9df3492abd48304a7f46ea4d778a5
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 20:49:52 2026 -0700
fix top_results adding in walk by filtering none results
commit 74746e6eca906b671bacf8eae6d29037aa4d086e
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 19:27:00 2026 -0700
update trestamps usage
commit 90e6a15f379c8b2d4d9578c4e2e456a04aee8742
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 16:00:16 2026 -0700
minor logging change
commit 484b730535c7b0dd6cdea90e9084a2e0df699fa8
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 15:43:37 2026 -0700
format and colorize doctor
commit 1de8b5c82830d5c4e2c441fdd28497e86d36cce1
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 15:34:22 2026 -0700
doctor mode cli help
commit cb36e7c22b011776d9804be86c707fa5722acebf
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 15:27:31 2026 -0700
version 6 and news
commit 009e61a5d57d49a112f4c26b30092566ccd84f08
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 15:24:40 2026 -0700
break up complexity
commit 97f486c0bd4e5a02df003bdb0f8ff7118a2ef88d
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 14:53:57 2026 -0700
clarify skipped container message
commit 48cd549b807d7aba8c75a333685159145233e078
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 14:35:39 2026 -0700
update deps
commit 3d487c76db3b7ea1ca39d2a21a63c0db3ba8304c
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 14:35:33 2026 -0700
fix mpo test
commit 25681f912016c54ca8dc876597acc54412d5d854
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 14:30:04 2026 -0700
fix MPO being counted as losssless
commit 96306e3c368ffae4822701dccb42ac256bd6f719
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 14:07:43 2026 -0700
don't clean computed config. needed for container in container
commit 4644ae1f48e48b31d42a7153819c52e16efce2b1
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 14:05:45 2026 -0700
move handler cleaning for repack into own function
commit 75ab979f050f10602051633f456a8c8c63a8a348
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 13:56:50 2026 -0700
remove illegal metadata arg from img2webp
commit d7310575400a5188621919660d88dde19b077e32
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 13:30:06 2026 -0700
move convert routes from rar & tar & sevenzip into zip
commit 10d3fd309e09ba9eecb9cb095f0e929feaac0ca4
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 12:51:30 2026 -0700
adjust some png conversion test sizes
commit 4f5377633315e93f9298521ed3b753a8443ee87f
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 12:51:05 2026 -0700
add webp routes for gif and png conversion
commit ba8ddc80d9123dca5ce0efc166e587fa55cc32de
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 12:11:33 2026 -0700
make a StdLibInternalTool for tar and zipfile
commit 6694c2b3f8f9595e79d7ce6fa565a9d49fbb1d2b
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 11:57:00 2026 -0700
consolidate tool base methods, version and path probing
commit 9512e724a226d8b47b3e525de744edc0b8804b33
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 11:24:29 2026 -0700
internal tool version parsing
commit 1bd799ff99fba1296bd31020f6346ede54649575
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 11:21:33 2026 -0700
internal tool version parsing. remove superfluous runtime typechecking code.
commit 9199b039908daac1ed56462baf4ee6a7c49179e2
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 11:20:36 2026 -0700
minor refactor
commit 049add42284d52be56e5beec84f1d170221f20c2
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 10:36:06 2026 -0700
move IS_WEBP_MOERN into CWebpTool from the handler. refactor doctor to be more self contained, use colors more modularly, and report cwebp version modernity inline rather than at the end.
commit cfc68fedb815babee9ac1a095f5f66bbe6a77bd5
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 00:44:56 2026 -0700
fix inheritance checking for import cycle
commit 7f53cc5072875f2c9fc9d329fa0633b1350c1379
Author: AJ Slater <aj@slater.net>
Date: Wed Apr 8 23:58:44 2026 -0700
new doctor and plugin architecture
commit 4f06ef75fccd510aef7b3b768e8c7b876099c66d
Author: AJ Slater <aj@slater.net>
Date: Wed Apr 8 23:57:51 2026 -0700
linted but many tests broken
commit ebb8decc00e1ae0ece2162e172cdc3f1ba078d74
Author: AJ Slater <aj@slater.net>
Date: Wed Apr 8 22:52:28 2026 -0700
format script
* add cancelled if to gha steps
* fix gha file cancelled() function
* install tool for tests
* update devenv
* remove circleci remnamnts
* format dockerfile
* add a set_do_repack() method for the upcoming scheduler to use
* for safety have buffered objects write to the tempfile before replacing the original.
* Scheduler (#103)
* add the scheduler first draft
* Replace walk/dispatch with futures-based scheduler (#101)
Swap multiprocessing.pool.Pool for concurrent.futures.ProcessPoolExecutor
and route all work through a new Scheduler that manages a ContainerNode
tree, three job types (Unpack/OptimizeLeaf/Repack), backpressure, and
per-node failure rollback.
Key changes:
- walk.py: delete _finish_results, _handle_container, _walk_container;
add _enqueue_children callback; walk() creates Scheduler and drains it
- init.py: Pool → ProcessPoolExecutor
- container.py: remove set_task/optimize_contents/_tasks; rename
_hydrate_optimized_path_info → hydrate_optimized_path_info (public API
for scheduler)
- archive.py: always copy unchanged files during walk (archive must be
open); adapt to new hydrate API; drop optimize_contents override
- pdf.py: rename hydrate method to match new public API
- scheduler.py: fix ContainerNode hashability (eq=False); use attribute
access for config; respect handler's own _do_repack flag for animated
image format conversion; delegate hydration to handler subclass
- config: add fail_fast and fail_fast_container options
- Bump treestamps ≥2.5.2 for pickleable GrovestampsConfig
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Add --fail-fast and --fail-fast-container CLI options (#102)
Expose the two new scheduler failure modes on the command line:
- --fail-fast: stop all optimization on the first error
- --fail-fast-container: fail the entire top-level container when
an inner repack fails
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* code cleanups"
* change method ordering for clairity. minor comment edits
* speling
* move old_timestamps into walk as legacy_timestamps
* refactor place format in plugins/base. make hanlder factory it's own class. consolidate report class and walk class.
* consolidate _handler-factory fucntion
* update deps
* code cleanup pass
* warn on digitial signature and encrypted pdf, do not error
* update deps
* use new treestamps methods
* webp tools superclass
* remove unused variable from pdf plugin
* update deps
* fix treestamps use and directory finishing, compacting
* v6.1.0 bump news
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* treestamps 3.0
* bump version and news
* update devenv. convert to bun
* Righttyper typing & corrections (#106)
* bump version and news
* rightyper typechecked
* Fix rightyper-introduced type annotation errors
Fixes 68 basedpyright errors and 51 ty errors introduced by the
automated rightyper pass. Changes are annotation-only.
- PathInfo.__init__ path_info parameter widened from None to PathInfo | None
- Removed spurious self: "picopt.xxx.Y" annotations naming wrong modules
- Simplified **kwargs bogus unions to Any for variadic super().__init__ passthroughs
- config/handlers.py handler_stages/convert_chain/native types widened to proper base types
- scheduler.py _trigger_fail_fast/_cancel_subtree widened Exception to BaseException | None
- detect_format.py filtered image.info keys to str; suppressed dynamic mpinfo attr
- pil_convertible _file_format image_class widened from BmpImageFile to Image
- tar.py removed type[TarGzDetector] cls annotation violating LSP
- tests: PosixPath -> Path; ts:None/config:None -> float|None/dict|None
- Cleaned unused TYPE_CHECKING imports left behind after removing bad annotations
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* correct typing changes
* revert to python 3.10 and remove righttyper
* update devenv
* remove quoted self types
* remove self types
* format & fix
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix bun install on docker and gha compose
* remove package-lock.json
* update devenv
* only copy bun deps first for dockerfile
* update devenv
* update devenv
* Fix BufferedReader pickling error when timestamps file exists (#107)
* fix pickling of ContainerHandler when a timestamps file exists
When a prior picopt run leaves a .picopt_treestamps.yaml, loading it via
ruamel.yaml leaves the YAML instance holding a Reader with an open
BufferedReader. Submitting an UnpackJob (or RepackJob) to the
ProcessPoolExecutor then fails with "cannot pickle 'BufferedReader'
instances" because the handler carries that Grovestamps through its
_timestamps / _skipper attributes.
Add __getstate__ on ContainerHandler that nulls both fields before
pickling. The scheduler owns the real Grovestamps on the main thread;
workers never actually used it (in-archive timestamp consumption
happened in a pickled copy whose mutations never propagated back).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* remove clean_for_repack, superseded by __getstate__
The method nulled the same two fields __getstate__ now nulls at pickle
time, and its one caller always ran on a handler that had already
round-tripped through a worker — so the fields were already None.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* print exception traces
* update deps
* bump news and version
* install poppler utils in docker for pdf tests
* update deps
* format dockerfile
* add bunx support
* bump version and news to v6.2.0
* try with svgo in packages
* Squashed commit of the following:
commit 83a5253785fccc471a6dbd75b4d1eba3074c9e8c
Author: AJ Slater <aj@slater.net>
Date: Wed Apr 29 02:18:19 2026 -0700
bump version and news
commit 9b03aed2141c50e3a088bc6c66bdc097cca7ced4
Author: AJ Slater <aj@slater.net>
Date: Tue Apr 28 20:44:04 2026 -0700
update treestamp loggins
commit 1b7bdc712edc80fc7cf67dc8d7c05f169462d764
Author: AJ Slater <aj@slater.net>
Date: Tue Apr 28 20:32:59 2026 -0700
fix implementation of timestamp loggins to log after not before
commit 8d25937c813ada944d120f3add9ae7b0852d6570
Author: AJ Slater <aj@slater.net>
Date: Tue Apr 28 20:25:28 2026 -0700
Log timestamp load/dump and surface INFO at default verbose=1 (#111)
Treestamps 4 dropped its own load/dump prints, and picopt's new logger
mapped INFO to verbose>=2, so default runs went silent for messages the
old termcolor Printer always showed (config-style force_verbose=True).
- _VERBOSE_LEVEL: bump so verbose=1 (the argparse default) emits INFO,
matching the old printer's "force_verbose" tier.
- walk.py: log "Loading timestamps for: …" before Grovestamps init and
"Dumping timestamps for: …" before dumpf(). INFO renders cyan via the
existing LEVEL_STYLES.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
commit 4415376f5f263cb6dedda5657765c745ca952f6f
Author: AJ Slater <aj@slater.net>
Date: Tue Apr 28 18:41:23 2026 -0700
Pre-walk file count for a determinate progress bar (#110)
* Pre-walk file count for a determinate progress bar
Without a total the bar showed only an indeterminate spinner + count.
Add ``Walk._count_total`` that mirrors ``walk_file``'s recursion gate
(symlinks, timestamp filenames, ignore patterns, recurse flag) so each
non-recursing visit contributes one mark — matching the events the
scheduler dispatches through Reporter — and pass it as ``total=`` to
``make_progress``.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Use os.scandir for the pre-walk count
DirEntry caches ``is_dir`` / ``is_symlink`` from the directory listing,
so deep trees skip an extra ``stat`` per entry — meaningful on slow or
network filesystems.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
commit 6a46d432777367a00ede8b183dfe8ff3bbf6a725
Author: AJ Slater <aj@slater.net>
Date: Tue Apr 28 17:48:59 2026 -0700
Replace termcolor Printer with loguru + rich logger (#109)
The old Printer class wrote dots and messages directly to stdout from
both the main process and worker processes. This change discards it for
a centralized logger modeled on nudebomb's progress branch:
- New picopt/log/ package: shared rich Console, loguru sink, a streaming
CharStreamColumn progress bar, a Stats + render() summary, and a
Reporter that bundles them and dispatches each ReportStats outcome.
- All call sites converted: Printer.{saved,converted,lost,error,skip,
warn,config,...} → logger.* and progress.mark_*. Worker-side dot and
lifecycle calls dropped — workers can't reach the parent's live region,
so per-file progress is now driven from the scheduler when each result
comes back.
- Centralized MARKS table in picopt/log/styles.py drives the streaming
chars, the loguru sink colors, the summary table row colors, and the
--help epilogue legend, so the same outcome reads identically
everywhere. Style choices mirror the old termcolor palette so longtime
users see the same colors for the same outcomes.
- Scheduler now takes a Reporter; Totals removed. report.py is a pure
data class (ReportStats) with no printer dependency.
- doctor.py and the cli help epilogue rewritten with rich (rich.markup
escape() for path strings).
- Grovestamps now constructed with verbose=0 so treestamps's internal
printer doesn't bypass the rich Live region.
- loguru~=0.7 added; termcolor dropped.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
commit 03cdcb018dce4049b6e4de54e71b50df33ddfd15
Author: AJ Slater <aj@slater.net>
Date: Tue Apr 28 17:34:02 2026 -0700
move set_jpeg_xmp into jpeg plugin
commit b28337465aacbad6a65808297dc4bf25115abfd7
Author: AJ Slater <aj@slater.net>
Date: Tue Apr 28 17:06:50 2026 -0700
update devenv. treestamps 4, rich 15
* fix pyright warnings
* ty ignore
* update treestamps
* fix MPO error for 6.3.1
* Migrate to confuse 2.2.0 by replacing AttrDict with PicoptSettings (#114) (#115)
Confuse 2.2.0 made ``AttrDict`` a properly-parameterized generic; an
unparameterized ``AttrDict`` resolves to ``AttrDict[str, object]``, so
every downstream access — ``cfg.bigger``, ``int(cfg.jobs)``,
``cfg.paths`` flowing into ``Iterable[str]``, etc. — now reads as
``object`` and stops type-checking. We had ~31 basedpyright errors after
the bump, all rooted at AttrDict consumers.
Fix: keep using confuse for what it's good at — YAML / env / CLI parsing,
type coercion, requiredness — and convert the validated ``AttrDict``
into a typed frozen dataclass once. Every downstream module then takes
``PicoptSettings`` instead of ``AttrDict``.
- ``picopt/config/settings.py``: new ``PicoptSettings`` (and nested
``ComputedSettings`` / ``IgnorePatterns``) frozen dataclasses mirroring
the existing ``MappingTemplate`` schema. ``Sequence(...)`` fields are
``tuple[X, ...]`` because the dataclass is frozen. ``handler_stages``
is typed ``dict[Any, Any]`` to avoid an import cycle with
``picopt.plugins.base``; consumers know its concrete shape.
- ``picopt/config/__init__.py``: ``PicoptConfig.get_config`` now returns
``PicoptSettings``. New ``_settings_from_attrdict`` does the
field-by-field copy (typed ``Any`` so it can read AttrDict attributes
directly — the conversion *is* the schema escape hatch).
- All consumers (``picopt/path.py``, ``picopt/report.py``, ``picopt/walk/*``,
``picopt/plugins/base/{handler,container}.py``) replace
``from confuse(.templates) import AttrDict`` with
``from picopt.config.settings import PicoptSettings``.
- ``Walk._init_timestamps`` builds a shallow ``Mapping[str, Any]`` for
Grovestamps' ``program_config=`` so we don't have to serialize the
whole frozen dataclass (especially the computed sub-fields, which
carry ``re.Pattern`` and class-keyed ``dict`` objects).
- ``is_path_ignored`` switched from ``ignore and bool(...)`` to
``ignore is not None and bool(...)`` — its return type is now
honestly ``bool``, not ``bool | re.Pattern | None``.
- ``Walk._dump_timestamps`` joins the dumpf return paths via ``str(p)``
(treestamps' ``dumpf`` returns ``tuple[Path, ...]``).
pyproject pin: ``confuse~=2.1.0,<2.2.0`` → ``confuse~=2.2.0``.
Verification: ruff clean, basedpyright clean (0 errors, was 31),
``uv run pytest --deselect …test_timestamp_parents`` 149 passed
(deselected test is a pre-existing treestamps v4 regression),
``radon cc`` all A/B, CLI smoke run with env var + CLI flag still
layers correctly.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Drop dead isinstance check; split Scheduler._submit_ready_job (#116)
Two warnings flagged by `make typecheck complexity`:
- ``picopt/config/__init__.py``: confuse 2.2.0 already types
``config.get(MappingTemplate)`` as ``AttrDict[str, object]``, so the
defensive ``isinstance(ad, AttrDict)`` check after it was statically
always-true (reportUnnecessaryIsInstance). Drop it and the now-unused
``AttrDict`` import.
- ``picopt/walk/scheduler.py``: ``_submit_ready_job`` was rank C in
radon. Split into three:
* ``_drop_cancelled_ready_job`` — counter cleanup for a leaf of a
cancelled subtree (UnpackJob/RepackJob skip; only leaf jobs
decrement the parent counter).
* ``_track_submitted_job`` — record an in-flight future under the
right map and update node state.
* ``_submit_ready_job`` — orchestrate: pop, cancelled-skip, submit,
track.
``_submit_ready_job`` is now rank A; the new helpers are A/B; behavior
unchanged.
Verified: ``make typecheck`` 0/0/0; ``make complexity`` no findings;
``uv run ruff check picopt/`` clean; ``uv run pytest`` 150 passed,
6 skipped.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* update deps
* bump version and news to 6.4.0
* update devenv
* Format argparse help with rich-argparse (#118)
Switch picopt's CLI to RawDescriptionRichHelpFormatter so the help
output is colorized. A PicoptHelpFormatter subclass adds one extra
highlight regex matching every registered format string (PNG, ZIP,
WEBP, …) under the `metavar` named group, so format names mentioned
inside help text share the color of their corresponding FORMATS /
EXTRA_FORMATS / CONVERT_TO metavars.
Rewrite the dot-color-key / doctor-mode epilog as Rich markup instead
of capturing ANSI from a Console. rich-argparse renders descriptions
and epilogs through `console.use_theme(Theme(self.styles))`, so the
epilog can use [argparse.groups], [argparse.prog], and [argparse.args]
to keep its section header, program name, and `doctor` subcommand
visually consistent with how the rest of the help is rendered.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* bump version and news to 6.5.0
* typecheck
* Defer animated WebP duration extraction to handler (#119)
detect_format eagerly iterated every frame of every animated image to
populate info["durations"], which only WebPMuxAnimatedLossless ever
consumed -- and that handler already had a webpmux-based fallback
(_read_durations) for when PIL's frame iteration was unreliable.
Removing the eager extraction speeds up format detection on animated
images (animated PNG ~11x, animated GIF ~4x in local microbenchmarks)
and simplifies the webp.walk path to always use the webpmux subprocess
fallback, which is the more reliable source per the original "PIL
frequently fails to populate per-frame durations on WebP" comment.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* bump news for animated deferall
* Cache file header bytes on PathInfo for detectors (#120)
PdfDetector was opening every file and reading 1024 bytes for every
detection attempt -- including non-PDF files where it would read,
fail the magic check, and return None. On a directory of non-image
non-archive files, this was ~14us per file (88% of the non-PIL
detector chain cost).
PathInfo now lazily reads up to 4 KB of file head into _header_bytes
on first access. PdfDetector consumes path_info.header_bytes() instead
of reopening the file.
Local benchmark of running the full non-PIL detector chain on
non-image files (pyproject.toml, README.md, uv.lock, Makefile):
~16us before -> ~2us after (8x).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* bump news
* Single Image.open in _extract_image_info; read metadata before verify (#121)
PIL's verify() closes its internal fp for some Path-opened formats
(notably GIF), which then breaks lazy attrs like is_animated and
n_frames. The previous code worked around this by opening the file
twice -- once to verify, again to read format/info.
Reading the lazy attrs before verify() preserves the safety check
(verify still runs and propagates corruption errors via the same
exception path) while collapsing to a single Image.open. Confirmed
on Pillow 12.2 against GIF (still + animated), TIFF (single + multi-
frame), MPO, animated PNG, animated WebP, JPEG, BMP, PPM.
_extract_image_info per-file (warm cache, microbenchmark):
test_animated_gif.gif 165us -> 95us
test_animated_png.png 145us -> 113us
test_animated_webp.webp 66us -> 30us
test_png.png 74us -> 35us
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* bump news
* update devenv and fix typechecking
* remove use of prettier sh plugin'
gi tpush
* fix data loss when converting without external tools
pil_save short-circuited format conversion when the input format was in
the handler's accepted set, even when the target format differed. This
caused PNG→WebP via PIL fallback to return the original PNG buffer
unchanged, which the cleanup code then deleted without writing a WebP.
Also fix pipeline selection ignoring tool.required=False, which caused
PNG optimization to be skipped entirely when pngout was absent despite
pyoxipng being available. Add PIL fallback for GIF when gifsicle is
missing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* doctor: show platform-specific install hints for missing tools
Detect the package manager (brew/apt/dnf) and display the install
command next to each missing external tool in `picopt doctor` output.
Also remove obsolete mozjpeg/jpegtran system install instructions from
the README — mozjpeg is now a bundled Python dependency that installs
automatically with picopt.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* decomplexify doctor
* update deps
* bump version and news
* update devenv. remove old prettier sh plugin
* replace pyoxipng with oxipng-pybind (#124)
Swaps the deprecated pyoxipng binding for oxipng-pybind, which wraps
oxipng v10.1.1 (vs v9.x) and is actively maintained. The module import
name is unchanged (import oxipng), so the only code change needed was
fixing one deprecated call: StripChunks.safe() → StripChunks.safe (enum
value, no parentheses). All 150 tests pass.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* update deps
* bump news and version to v6.5.2
* update deps
* update deps
* Add memory-aware scheduling to prevent OOM on large archives
Optimizing large archives (e.g. multi-GB comic books) could exhaust RAM
and get the process OOM-killed. Each container holds its whole archive,
all decompressed members, the optimized copies, and the output buffer in
memory at once, replicated into each parallel worker, and nothing bounded
the in-flight total beyond a future-count cap.
Give the scheduler a memory budget: estimate each top-level archive's peak
resident memory (~3x its on-disk size) and only admit a new one while it
fits the budget, with a forward-progress guarantee so a single archive
larger than the whole budget still runs (alone). The gate scans past
budget-blocked items so in-flight work still frees the budget, avoiding
both deadlock and head-of-line blocking.
Add a --memory-limit option (e.g. 8G, 512M) that sets the budget as an
approximate peak-memory target; the default is two-thirds of detected RAM.
It is excluded from the timestamps config so it never invalidates stamps.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* update deps and format
* Fix basedpyright and ty diagnostics
Convert mypy-style `# type: ignore[arg-type]` comments in the scheduler
memory test to the project's paired `# pyright: ignore # ty: ignore`
convention (neither checker honors mypy-style comments), and make the
`_charge_info` match exhaustive with an explicit `case _` for the
intentionally-exempt RepackJob path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* update devenv
* remove pytest-gitignore
* Fix critical data-loss bugs in file replacement and archive repacking
- Write optimized results to a sibling temp file and atomically replace
the original; a crash, OOM-kill, or full disk mid-write can no longer
destroy the user's file.
- Keep members whose optimization errored (and nested containers whose
repack failed) in the repacked archive instead of silently dropping them.
- Open nested archives from their in-memory buffer; archives inside
archives previously always failed to unpack with FileNotFoundError.
- Discarded conversion results (bigger output, dry-run) no longer stat a
never-written path and are reported as skipped instead of errors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Reduce _select_pipeline_for_handler cognitive complexity
Extract per-tier tool picking into _pick_tier_tool and drop the
redundant empty-pipeline guard; behavior is unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Fix scheduler and walk error-path correctness
- A member that fails format detection (e.g. a decompression bomb) is
recorded as one error and passed through to the repacked archive
instead of aborting the entire run and losing all timestamps.
- Directories and containers with errored children no longer get
timestamps; previously the compacted dir stamp covered failed files
so they were silently skipped on every subsequent run.
- Failed top-level containers now notify their directory tracker, which
previously leaked and blocked ancestor timestamp finalization.
- Nested container repacks hydrate through the parent handler so
conversion renames (e.g. inner CBR -> CBZ) are no longer lost.
- Requested container conversions repack even when every member was
already optimized; CBR -> CBZ previously did nothing in that case.
- Ignore patterns now respect filesystem case sensitivity (the flag
was inverted), timestamps files are matched by basename, and FIFOs/
sockets are skipped instead of blocking the walk forever.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Preserve archive metadata fidelity through repack
- Archive comments (zip, rar) are read during walk and written back on
repack; _set_comment previously had no call site so comments were
always destroyed (CBZ comments commonly hold ComicBookInfo tags).
- Repack preserves original member order via a walk-time archive_index
on PathInfo; the optimized-contents set previously scrambled order,
which broke EPUB's mimetype-first requirement and reproducibility.
- Member times before 1980 (or missing) are clamped to the zip epoch
instead of crashing zipfile on conversion.
- Converted members choose compression by content: text deflates,
already-compressed image formats are stored. Previously rar/tar/7z
conversions wrote every member uncompressed, and --strip-metadata
wastefully deflated incompressible images.
- Legacy zip member names that are really UTF-8 misread as CP437 are
repaired at repack instead of being re-encoded as mojibake forever.
- Tar links survive repack: symlinks/hardlinks keep their type, target,
and zero size; link targets are no longer duplicated into link
entries (or crashes on external targets).
- 7z member mtimes are restored after repack; py7zr's writef API stamps
members with now() and offers no public override.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Fix config layering, validation, and doctor accounting
- Config-file and env values for boolean options and verbose are no
longer silently shadowed: un-passed CLI flags now default to None,
which confuse drops, so lower layers stay visible. CLI flags that ARE
passed still override everything.
- -I/--no-default-ignores without -i no longer aborts at startup (the
computed ignore patterns are validly None).
- Unparseable --memory-limit values abort with a clean config error
instead of silently falling back to auto, and unparseable --after
values no longer dump a raw dateutil traceback. Timezone-aware
--after values are interpreted correctly instead of as local time.
- picopt doctor counts requirements per pipeline tier instead of per
tool; tools within a tier are alternatives, so healthy installs no
longer report missing-required tools and exit 1.
- Lowercase format lists are accepted for -x and -c exactly as for -f.
- README no longer documents the removed PICOPT_ENABLE_IMG2WEBP
environment variable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Fix webp lossless detection, png depth parsing, and handler edge cases
- is_lossless() parses the RIFF chunk structure instead of searching
for byte strings. The old code searched the whole file via mmap for
filesystem input (metadata bytes could fake losslessness) but only
128 bytes for in-memory input (a large ICC/EXIF chunk before VP8L
misdetected lossless in-archive WebPs as lossy). It also closed the
caller's buffer and crashed mmap on empty files.
- png_bit_depth() returns None for truncated PNGs instead of a fake
0 depth.
- run_ext_fs() cleans up spilled temp files when the external tool
fails instead of leaking them.
- --preserve survives chown permission failures (non-root users) and
still restores mode and mtime instead of reporting the whole
optimization as an error.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Make PDF and SVG optimization safer
- Repacking preserves PDF encryption: owner-password-restricted PDFs
open without a password and were silently saved decrypted. qpdf
cannot decode streams while preserving encryption, so encrypted PDFs
get the weaker (but protected) rewrite.
- Optimized JPEGs are written back with their original /DecodeParms
(e.g. /ColorTransform), which pikepdf's write() otherwise drops.
- Signature detection also looks for /Type /Sig objects, catching
signed PDFs from producers that never set the AcroForm SigFlags bit.
- svgo runs with an explicit config instead of the default preset:
removeViewBox is disabled (it breaks responsive scaling), and title/
desc/metadata elements follow the --strip-metadata flag instead of
the preset's whims.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Survive symlink loops and unreadable directories
- walk_dir, the progress pre-count, and the legacy timestamp import
track visited directories by (st_dev, st_ino); symlink cycles and
duplicate links are walked once instead of recursing until the
filesystem errors (or, in the pre-count, forever).
- The legacy timestamp import skips unreadable paths with a warning
instead of crashing the whole run at startup, and matches ignore
patterns with the same case-sensitivity fix as the walk skipper.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Revive in-archive entry skipping and timestamp consumption
ContainerHandler.walk() always executes in a worker process, but
__getstate__ dropped the Grovestamps (unpicklable ruamel state) it
needs, so member entry-mtime skipping, in-archive .picopt_treestamps
consumption, and the timestamps_ignore_archive_entry_mtimes option
were all dead code.
Archive handlers now carry an ArchiveStamps slice instead: the tree's
picklable config plus its dump_dict() serialization (the same
config-headed mapping treestamps round-trips through disk). A real,
detached Treestamps is rebuilt lazily in the worker — never set() or
dumped, so it never touches the WAL — preserving full treestamps
semantics: ancestor-max lookups and config-checked loading of stamp
files found inside archives.
Members whose entry mtimes predate the archive's stamp are now
noop-copied instead of re-optimized when a stamped archive is
modified, stamp files inside archives are consumed and dropped on
repack, and -E disables entry-mtime skipping as documented.
timestamps_ignore_archive_entry_mtimes joins TIMESTAMPS_CONFIG_KEYS
since flipping it changes which members get optimized.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Improve scheduler-thread and archive throughput
- Archive member format detection (PIL open + verify per member) moves
into the unpack workers via predetect_format(), which caches onto
PathInfo; the scheduler thread previously decoded every member of
every archive serially.
- The progress pre-count no longer re-walks the whole tree when the
progress bar is disabled (-q).
- Top-level results no longer pickle their optimized bytes back to the
main process just to be discarded; only in-container entries return
data for their parent's repack.
- Items deferred by the memory gate wait in a FIFO gated queue retried
head-first per tick instead of rescanning the entire ready queue on
every completion (O(queue) per event under memory pressure).
- 7z archives extract all members in one pass; solid archives
previously decompressed from the start for every single-target
extract, O(n^2) over members. The extraction cache is dropped before
the handler pickles back to the main process.
- Add bin/benchmark.py: wall-clock + per-run peak RSS harness over
synthetic fixture corpora, for measuring future changes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Extract dir timestamping, wire staging cleanup, untangle reporter dispatch
- The per-directory completion tracking and compacted timestamp writes
move from the scheduler into walk/dir_timestamps.DirTimestamper; the
scheduler delegates its begin/seal/cancel API.
- All five container pending-counter decrements route through one
_child_done() method; _maybe_start_repack's state guards make it safe
on every completion path.
- The staging-dir cleanup design is finally wired: handlers expose
get_staging_dir(), the scheduler stores it on the node after unpack,
and rollback/fail-fast/final cleanup actually remove worker tmp dirs.
Animated-WebP frame directories previously leaked on cancellation.
- Reporter outcome dispatch is an explicit match instead of
getattr(f"record_{kind}") string dispatch that failed silently on
renames.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Split the webp plugin into a package
plugins/webp.py (759 lines, five handlers, four pack strategies) becomes
plugins/webp/: const, tools, static, and animated modules with the
PLUGIN descriptor in __init__. Plugin discovery now accepts subpackages.
The CWebPTool.probe() -> IS_MODERN_CWEBP class-attribute side effect is
replaced by an is_modern flag on the shared, probed-once CWEBP_TOOL
singleton that WebPLossless's PIPELINE holds and both consuming handlers
read at construction time. The tools no longer import handler classes at
all — they duck-type on the args-provider methods — removing the
tool/handler import cycle the isinstance checks required.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Deduplicate handler routing and document the PathInfo contract
- plugins.pick_route_handler is the one routing decision: the factory
uses it at runtime and config/handlers uses it to build the startup
formats summary, replacing an 8-parameter re-derivation whose logic
had already drifted from the factory's (it ignored empty-pipeline
availability). The log can no longer disagree with the walk.
- PathInfo's class docstring documents the mutable-identity contract:
instances cross the process boundary and are mutated by the scheduler
after hydration, with cache invalidation owned by rename()/set_data().
- computed.handler_stages is structurally typed dict[type, tuple];
the concrete Handler/Tool annotation is impossible because
plugins.base imports settings and basedpyright's reportImportCycles
counts even TYPE_CHECKING edges.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Memoize tool probes
probe() caches its ToolStatus per instance and concrete probes move to
_probe(). Tools are module singletons, so each external tool spawns its
--version subprocess once per process — a prerequisite for per-directory
config files, whose confuse rebuilds rerun the pipeline probe loop.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Add per-directory .picopt.yaml config files and config-writing flags
Ported from nudebomb's implementation (same confuse + treestamps stack).
A .picopt.yaml placed in a target directory applies to that directory
and everything beneath it. Files are discovered from each processed
file's directory up to the CLI target root (never above it); deeper
directories win over shallower ones, and env vars and CLI options still
win over every directory file — implemented by rebuilding a fresh
confuse Configuration per unique directory chain through the extracted
_build_config layering core. DirConfig caches resolutions per directory
with an identity fast path when no config applies, and isolates
malformed files (logged once, recorded in stats, fall back to run-wide
settings). Run-mode keys (dry_run, list_only, timestamps) are pinned to
run-level values: a per-directory dry run with run-level timestamps
would stamp files that were never optimized.
Per-directory settings reach everything: guards (ignore/symlinks/
recurse) via per-directory skippers and the pre-count mirror, handlers
via the factory's settings parameter (formats/convert_to/bigger/
keep_metadata/... travel into workers), archive members by inheriting
their container's config (nesting inherits automatically), and repacks
run under the handler's own config. .picopt.yaml files are skipped as
optimization targets by basename.
With -t, a sha256 fingerprint of every .picopt.yaml under the targets
joins the treestamps program-config check, so editing, adding, or
removing a directory config re-processes its tree (over-invalidation,
never a wrong-skip).
New write flags, all validate-before-persist with round-trip YAML
merges that preserve comments and unrelated keys, written owner-only:
-w/--write-config (user config), -W/--write-dir-config (a .picopt.yaml
in each target directory), and --write-config-file PATH. Run-mode
flags and target paths are never persisted.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Add fail-fast, cancellation, idempotency, and pixel-identity tests
- --fail-fast cancels every live top-level container and clears both
scheduler queues on the first repack error; --fail-fast-container
escalates an inner failure to its subtree root while leaving sibling
top-level containers untouched; a leaf result arriving after its
parent's cancellation is dropped without touching the parent's
contents.
- Optimization converges: a second pass may only shrink files (the
multi-tool png pipeline legitimately shaves another byte when one
tool's output enables another) and every file reaches its fixed
point by the third pass.
- Lossless paths are pixel-exact: png optimization, png-to-webp, and
gif-to-png conversions decode to identical RGBA pixels.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Update NEWS for v6.6.0
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Add missing v6.6.0 news items
Fill in five user-facing fixes that landed since v6.5.2 but were absent
from the v6.6.0 entry: archive conversions recompress members by content,
--preserve survives non-root chown failures, temp files and animated-WebP
frame dirs are cleaned up on failure/cancellation, timezone-aware --after
is honored, and -I without -i no longer aborts at startup.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* version 6.6.0
* Fix repeated "Optimizing formats" banner with -W and planted .picopt.yaml
The verbose "Optimizing formats" banner is a run-wide summary, but it was
emitted from set_format_handler_map(), which _build_config() re-runs for
every per-directory .picopt.yaml resolution during the walk. With -W (or
any planted directory config in the tree root), every directory finds a
config in its ancestry, skips the "reuse run-wide settings" fast path, and
rebuilds the config — reprinting the banner once per directory.
Gate the banner behind a print_summary flag so only the top-level
get_config() build emits it. Move the summary computation into
_log_formats_summary() so the per-directory rebuilds skip it entirely.
Add a regression test asserting the banner prints exactly once under -W.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* bump version to 6.6.1
* bump news
* Fix animated WebP optimization failing on zero-duration frames
img2webp rejects any frame duration <= 0 ("Invalid negative duration"),
aborting the whole animated-WebP repack. Some sources (e.g. scraped
animated WebPs) carry 0ms frame durations, so clamp each duration up to
img2webp's 1ms minimum in Img2WebPAnimatedLossless.img2webp_args.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* update deps
* Clear basedpyright typecheck warnings
- Use a real PicoptSettings (not SimpleNamespace) in the img2webp
duration test so config assignment type-checks.
- Mark the module-level pytest autouse fixtures with
`# pyright: ignore[reportUnusedFunction]`; they are used by pytest,
not referenced by name.
- Disable reportImplicitStringConcatenation: ruff format canonicalizes
adjacent string literals to implicit concatenation, so the check can
never be satisfied.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix news formattign
* Reduce cyclomatic complexity of four rank-C functions
radon cc flagged four functions at rank C. Extract behavior-preserving
helpers to bring each to rank A/B:
- WalkSkipper._is_skippable (16): split into _skip_reason and the
disk-only _skip_reason_fs.
- Walk._count (12): extract the recursion-stop predicate _count_stops_here.
- pick_route_handler (12): fold the convert gate into _pick_convert_handler
and extract _can_pack_repack for the repack check.
- Handler._cleanup_after_optimize (11): extract the _should_replace predicate.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Print verbose config summaries only once, not per directory config
Per-directory .picopt.yaml resolution rebuilds the confuse config for
every directory in the tree, repeating the timestamps, ignore, after,
and memory-limit log lines each time. Gate them on the existing
print_summary flag, as set_format_handler_map already does.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* update deps and bump version to 6.6.2
* Add v6.6.2 news entry for repeated verbose summaries fix
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix timestamps silently invalidated by tree-root .picopt.yaml configs
The treestamps program config was recorded from the run-level CLI
config, so identical options expressed via a tree root's .picopt.yaml
instead of flags always mismatched and every stamp was silently
discarded, re-optimizing whole trees.
Each tree now records its own effective config: the new Grove builds
one Treestamps per top path from DirConfig.get_tree_settings, which
resolves the root's .picopt.yaml beneath CLI and env. timestamps: true
at a tree root now enables stamping without -t (dry run and list still
disable it, and subdirectory configs still can't toggle it mid-walk).
The directory-config fingerprint is now per tree and value-based: it
hashes the parsed picopt section of each config strictly below the
root, so comment and formatting edits, -W rewrites, running with a
different set of top paths, and file-vs-directory targets of the same
tree no longer invalidate stamps — only real option changes do. The
root's own config is excluded because its options are recorded as
values.
Stamps discarded on a config mismatch now warn with the differing keys
(treestamps 4.1.1); stdlib logging is intercepted into loguru so the
warning renders through the shared console, and log messages escape
Rich markup so bracketed names like "Title [en]" display intact.
Migration: existing stamp files mismatch once after this upgrade (now
loudly); a single -N run preserves old stamps instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T7R5s6frL7jEykysAzjttq
* update deps
* Report corrupt images as clean warnings instead of dumping tracebacks
PIL raises a plain OSError when it recognizes an image format but the
contents are corrupt (e.g. a BMP with pixel depth 0), which escaped the
detection probe and printed a full stack trace. Detection now wraps it
in UnreadableImageError, reported as a one-line warning that is also
counted and listed in the summary; the file is skipped untouched.
The per-file exception guards now skip the stack trace for any
PicoptError — extending the contract cli.main already had: expected
conditions print their message, only unexpected exceptions dump stacks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T7R5s6frL7jEykysAzjttq
* version 6.6.3
* update deps
* Fix ty and remark-lint warnings
- Drop the `cls: type[Any]` annotation on the RarDetector/ZipDetector
`identify` classmethods; ty flagged the widened first parameter as an
invalid override of `Detector.identify`.
- Replace remark's `no-duplicate-headings` with
`no-duplicate-headings-in-section`. A changelog necessarily repeats
`### Fixes` under each version; the in-section rule still catches real
duplicates.
Also includes the in-progress dependency bumps and ruff rule additions that
were already in the working tree.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Jxl format (#129)
* Add JPEG XL support
Read and write JPEG XL via the pillow-jxl-plugin codec, as a new
auto-discovered plugin. JXL is optimized by default and is now the
preferred still conversion target ahead of WEBP and PNG.
Only lossless JXL is re-encoded; lossy (XYB encoded) JXL is left alone,
the same policy lossy WebP already has. The codec exposes no such signal,
so picopt/pillow/jxl.py parses the codestream image header to read
xyb_encoded, and refuses files carrying JPEG reconstruction data, whose
re-encoding would silently destroy the ability to restore the source JPEG.
New --convert-jpeg-to-jxl converts JPEG to JXL by lossless bitstream
reconstruction, which is reversible byte for byte. It requires -c JXL as
well: picopt otherwise never converts from a lossy source, so -c JXL alone
must not sweep up a whole photo library. The codec only takes its
reconstruction path when PIL knows the source filename, which it does only
for path opens, so the tool opens by path and spills to a temp file for
inputs that have none. That makes it work for archive members too.
Three smaller things the codec needed:
- It registers no save_all handler, and pil_save always passes save_all,
which raised KeyError. The single-frame handler is registered for it;
animated JXL is unsupported by the codec regardless.
- Given no exif kwarg it reads EXIF off the opened image, which quietly
defeated --strip-metadata. prepare_info now passes an empty exif.
- It rejects palette and other modes, so ImageHandler gained a
prepare_image hook, symmetric with prepare_info, to substitute one.
Handlers can now declare CONFIG_ENABLED_KEY, a config flag that must be
true for them to be probed at all. Failing it leaves the handler out of
the probed handler map, which routing already reads as unavailable, so the
startup conversion banner stays accurate for free.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Gate WebP to JXL conversion behind --convert-webp-to-jxl
JPEG XL support in browsers and image tooling is still thin, so converting
away from WebP, which every browser reads, should be a deliberate act
rather than a side effect of asking for -c JXL.
JPEG and WEBP are both formats picopt processes by default, which means
the usual "name the source with -x" gate cannot express a choice about
them: -x JPEG and -x WEBP are no-ops. Each therefore carries its own flag,
and a plain -c JXL once again converts only GIF and PNG, as the README has
always described. The previous commit had quietly made WEBP a third
convert-by-default source.
The flag hangs off the handler class, and JxlLossless also serves PNG, GIF
and native JXL, which must stay ungated. So the WebP route now points at a
JxlFromWebP subclass that exists only to carry CONFIG_ENABLED_KEY.
Converting out of JXL into a more widely readable format stays ungated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* update deps
* update deps
* update deps
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* update deps
* v6.8.0: timestamps survive upgrades that record new options
Adding an option to the recorded timestamps config used to invalidate
every existing timestamp file, re-optimizing whole trees. Picopt now
passes its packaged defaults to treestamps 5.0.0, which fills keys
missing from either side before comparing, so only a changed option
value invalidates. Timestamp files written before 6.7.0 added the JXL
options, and before 6.6.2 added the directory config fingerprint, are
honored again.
Timestamp files now open with a comment saying what they are, that they
are machine-written, and that deleting one re-optimizes that tree. The
redundant treestamps_config block is gone; older files still load.
Discarding timestamps because a sub-directory .picopt.yaml changed now
names it that way instead of printing the internal fingerprint key.
Replace the local directory config fingerprint with the identical
treestamps implementation, pinned by golden digests upstream.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* update deps
* Write config files atomically
merge_config_file truncated the target before writing, so a crash, a
full disk, or a kill during -w, -W, or --write-config-file could
destroy a hand-authored .picopt.yaml or user config. Write to a unique
sibling temp file, fsync, then replace, matching the discipline the
image handler and treestamps already use for files that matter less.
The temp name is unique rather than a fixed sibling so two concurrent
writers cannot unlink each other's temp mid-write, and the owner-only
mode is set before the rename so the target is never briefly wider.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Note the atomic config writes in the v6.8.0 news
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
ajslater
added a commit
that referenced
this pull request
Aug 3, 2026
* configure mkdocs properly
* tilde versions for deps
* add html section to readme
* update deps
* update deps
* revise project urls
* update deps add ty. typecheck with ty
* uv_build
* update scripts, pyproject, eslint, depts, fix lints
* update deps
* update deps ane eslint config
* update deps
* doesn't work with python 3.14
* fix builder dockerfile
* copy debian sources on build properly for ci
* format
* fix tests for new versions of external programs on darwin and somewhat on linux
* fix formatting and use python 3.13 for build
* format
* fix linux test file sizes
* more test size changes for linux
* remove test subdirs
* fix webp sizes in tests for linux
* try to fix pre optimized webp on linux again
* fix sizes again
* one more size
* update deps
* update deps
* add ty to makefile
* delete deprecated pillow code
* support python 3.14, remove old pyright ignores
* bump version and news
* update deps
* update deps
* update deps
* try to get duration info from detect format and PIL
* use webpmux for webp animated
* update deps, update pytest config
* change container test names
* fix PIL jpeg conversion size for mpo test
* replace old test animated webp with one without dropped frames
* enable im2webpanimatedlossless permanently enable webpumx, adjust file sizes so image_dir tests all pass. container tests still not passing
* fix adding info to correct handler clases constructor
* add icecream importer
* bump news and version
* more news
* simplify run_ext_fs for webp no longer needing output_path. compact and simplify creating working paths
* change webp arg order
* use python secure tmpdir for webp temporary files
* fix news version number
* fix test webp sizesfor linux
* update deps
* fix method and variable speling
* type checking fixes
* reduct complexity
* complexipy & dep groups
* reduce complexity of create handler even more
* clean now cleans all caches
* update deps
* fix image format detection
* isolate handler external command tuples for debugging
* fix extracting info from images
* fix conversion to 7zr
* update deps
* remove unused ty ignores
* ty ignores
* update deps
* update deps
* update deps
* update deps
* turn some const sets into tuples for type compatibility
* use tuple instead of list
* type get_config method. remove useless warning raise
* update deps
* fix attrdict type
* update deps
* fix mpo test
* bump version and pin confuse, bump news
* fix test script with test dependency group
* fix lint script with group
* change confuse version specifier
* use dockerfmt. update deps. ruff converts inline ifs to ors
* update build dockerfile
* update deps and add eslint plugins
* add mbake
* remove unused ruff noqas
* Add types with pyre-check
* infer types. bump version and news. add py.typed
* Squashed commit of the following:
commit b7bedc7813be90cb8e922d2dc4fbd3ca55b77b81
Author: AJ Slater <aj@slater.net>
Date: Sat Feb 14 18:26:52 2026 -0800
update devenv
commit 117c504caf8fcd2590e45731b9c4f38c74d5d9bc
Author: AJ Slater <aj@slater.net>
Date: Sat Feb 14 14:45:09 2026 -0800
update devenv scripts
commit 22e7f164469599e98848862589a52f2311ad4e4e
Author: AJ Slater <aj@slater.net>
Date: Sat Feb 14 14:41:27 2026 -0800
add fix merged to update devenv
commit 738ed22be3f3d75c1961339c8921e4d09d7466b8
Author: AJ Slater <aj@slater.net>
Date: Sat Feb 14 14:36:17 2026 -0800
update devenv
commit 848e9911ecc05af9122a1e361755d21925cbcb85
Author: AJ Slater <aj@slater.net>
Date: Sat Feb 14 11:01:58 2026 -0800
remove init script
commit 762550dc406c2345b430833acc961860515c98df
Author: AJ Slater <aj@slater.net>
Date: Sat Feb 14 11:00:08 2026 -0800
update dev scripts
commit 2bc0d768c2c20a30ed1d2f67495002315483f30f
Author: AJ Slater <aj@slater.net>
Date: Sat Feb 14 01:30:49 2026 -0800
use devenv
* update devenv and deps
* update devenv
* update devs and devenv
* update devenv
* add option for ignore archive entry mtimes
* change option name
* update devenv add missing ci script
* fix script hashbang
* move docker-compose to compose remove ignored version
* try running script with simpler sh
* add debug command for circlecir persist
* chown dist before store
* speling
* update devenv
* remove cruft
* update makefile
* update deps
* update devenv
* update devenv
* update devenv
* update devenv
* update devevn
* update devenv
* update devenv & deps
* update deps
* update devevn & deps
* Squashed commit of the following:
commit 5fca5a970e8cf0e1184fd5aae6bd50dd6445a75c
Author: AJ Slater <aj@slater.net>
Date: Wed Apr 8 15:32:09 2026 -0700
disable circleci switch to gha
commit 9f1b3f993f87c8a94eedf8138f14be06def52724
Author: AJ Slater <aj@slater.net>
Date: Wed Apr 8 15:27:18 2026 -0700
skips writing timestamps if no other disk writing operation occurred or needs to
commit 2d40924553cd525696381b04509505eb39e7bed1
Author: AJ Slater <aj@slater.net>
Date: Wed Apr 8 11:46:10 2026 -0700
use env for main picopt test script
* dockerfile becomes builder
* update devenv
* Squashed commit of the following:
commit c82831c17adbf2ca6e44b8fe2734235d8830e6fb
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 23:42:05 2026 -0700
lots of code cleanups mostly match instead of if then chains and Path consistency. One big PathInfo copy constructor that's a little too clever
commit be01ee405c37e19c590f0a667c13775b3b58b6de
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 22:46:28 2026 -0700
minor cleanups for pillow patches
commit a590f6dfea20e800018a900bbf2370f15f3872c5
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 22:42:35 2026 -0700
pdf plugin and tests with variance allowances
commit a6e81ebef927b1c8e4f7c4775895983668f5ebac
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 22:24:01 2026 -0700
add pdf tests
commit 1aa105d2d93791e608119cc95d5555d7d144e866
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 22:05:39 2026 -0700
PDF support
commit 732a14a7629a2cf829d8e2e410cb6d5004900b4d
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 20:50:53 2026 -0700
fix changed calculation for in memory optimization
commit 962fbc9ab9d9df3492abd48304a7f46ea4d778a5
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 20:49:52 2026 -0700
fix top_results adding in walk by filtering none results
commit 74746e6eca906b671bacf8eae6d29037aa4d086e
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 19:27:00 2026 -0700
update trestamps usage
commit 90e6a15f379c8b2d4d9578c4e2e456a04aee8742
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 16:00:16 2026 -0700
minor logging change
commit 484b730535c7b0dd6cdea90e9084a2e0df699fa8
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 15:43:37 2026 -0700
format and colorize doctor
commit 1de8b5c82830d5c4e2c441fdd28497e86d36cce1
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 15:34:22 2026 -0700
doctor mode cli help
commit cb36e7c22b011776d9804be86c707fa5722acebf
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 15:27:31 2026 -0700
version 6 and news
commit 009e61a5d57d49a112f4c26b30092566ccd84f08
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 15:24:40 2026 -0700
break up complexity
commit 97f486c0bd4e5a02df003bdb0f8ff7118a2ef88d
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 14:53:57 2026 -0700
clarify skipped container message
commit 48cd549b807d7aba8c75a333685159145233e078
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 14:35:39 2026 -0700
update deps
commit 3d487c76db3b7ea1ca39d2a21a63c0db3ba8304c
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 14:35:33 2026 -0700
fix mpo test
commit 25681f912016c54ca8dc876597acc54412d5d854
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 14:30:04 2026 -0700
fix MPO being counted as losssless
commit 96306e3c368ffae4822701dccb42ac256bd6f719
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 14:07:43 2026 -0700
don't clean computed config. needed for container in container
commit 4644ae1f48e48b31d42a7153819c52e16efce2b1
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 14:05:45 2026 -0700
move handler cleaning for repack into own function
commit 75ab979f050f10602051633f456a8c8c63a8a348
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 13:56:50 2026 -0700
remove illegal metadata arg from img2webp
commit d7310575400a5188621919660d88dde19b077e32
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 13:30:06 2026 -0700
move convert routes from rar & tar & sevenzip into zip
commit 10d3fd309e09ba9eecb9cb095f0e929feaac0ca4
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 12:51:30 2026 -0700
adjust some png conversion test sizes
commit 4f5377633315e93f9298521ed3b753a8443ee87f
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 12:51:05 2026 -0700
add webp routes for gif and png conversion
commit ba8ddc80d9123dca5ce0efc166e587fa55cc32de
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 12:11:33 2026 -0700
make a StdLibInternalTool for tar and zipfile
commit 6694c2b3f8f9595e79d7ce6fa565a9d49fbb1d2b
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 11:57:00 2026 -0700
consolidate tool base methods, version and path probing
commit 9512e724a226d8b47b3e525de744edc0b8804b33
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 11:24:29 2026 -0700
internal tool version parsing
commit 1bd799ff99fba1296bd31020f6346ede54649575
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 11:21:33 2026 -0700
internal tool version parsing. remove superfluous runtime typechecking code.
commit 9199b039908daac1ed56462baf4ee6a7c49179e2
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 11:20:36 2026 -0700
minor refactor
commit 049add42284d52be56e5beec84f1d170221f20c2
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 10:36:06 2026 -0700
move IS_WEBP_MOERN into CWebpTool from the handler. refactor doctor to be more self contained, use colors more modularly, and report cwebp version modernity inline rather than at the end.
commit cfc68fedb815babee9ac1a095f5f66bbe6a77bd5
Author: AJ Slater <aj@slater.net>
Date: Thu Apr 9 00:44:56 2026 -0700
fix inheritance checking for import cycle
commit 7f53cc5072875f2c9fc9d329fa0633b1350c1379
Author: AJ Slater <aj@slater.net>
Date: Wed Apr 8 23:58:44 2026 -0700
new doctor and plugin architecture
commit 4f06ef75fccd510aef7b3b768e8c7b876099c66d
Author: AJ Slater <aj@slater.net>
Date: Wed Apr 8 23:57:51 2026 -0700
linted but many tests broken
commit ebb8decc00e1ae0ece2162e172cdc3f1ba078d74
Author: AJ Slater <aj@slater.net>
Date: Wed Apr 8 22:52:28 2026 -0700
format script
* add cancelled if to gha steps
* fix gha file cancelled() function
* install tool for tests
* update devenv
* remove circleci remnamnts
* format dockerfile
* add a set_do_repack() method for the upcoming scheduler to use
* for safety have buffered objects write to the tempfile before replacing the original.
* Scheduler (#103)
* add the scheduler first draft
* Replace walk/dispatch with futures-based scheduler (#101)
Swap multiprocessing.pool.Pool for concurrent.futures.ProcessPoolExecutor
and route all work through a new Scheduler that manages a ContainerNode
tree, three job types (Unpack/OptimizeLeaf/Repack), backpressure, and
per-node failure rollback.
Key changes:
- walk.py: delete _finish_results, _handle_container, _walk_container;
add _enqueue_children callback; walk() creates Scheduler and drains it
- init.py: Pool → ProcessPoolExecutor
- container.py: remove set_task/optimize_contents/_tasks; rename
_hydrate_optimized_path_info → hydrate_optimized_path_info (public API
for scheduler)
- archive.py: always copy unchanged files during walk (archive must be
open); adapt to new hydrate API; drop optimize_contents override
- pdf.py: rename hydrate method to match new public API
- scheduler.py: fix ContainerNode hashability (eq=False); use attribute
access for config; respect handler's own _do_repack flag for animated
image format conversion; delegate hydration to handler subclass
- config: add fail_fast and fail_fast_container options
- Bump treestamps ≥2.5.2 for pickleable GrovestampsConfig
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Add --fail-fast and --fail-fast-container CLI options (#102)
Expose the two new scheduler failure modes on the command line:
- --fail-fast: stop all optimization on the first error
- --fail-fast-container: fail the entire top-level container when
an inner repack fails
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* code cleanups"
* change method ordering for clairity. minor comment edits
* speling
* move old_timestamps into walk as legacy_timestamps
* refactor place format in plugins/base. make hanlder factory it's own class. consolidate report class and walk class.
* consolidate _handler-factory fucntion
* update deps
* code cleanup pass
* warn on digitial signature and encrypted pdf, do not error
* update deps
* use new treestamps methods
* webp tools superclass
* remove unused variable from pdf plugin
* update deps
* fix treestamps use and directory finishing, compacting
* v6.1.0 bump news
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* treestamps 3.0
* bump version and news
* update devenv. convert to bun
* Righttyper typing & corrections (#106)
* bump version and news
* rightyper typechecked
* Fix rightyper-introduced type annotation errors
Fixes 68 basedpyright errors and 51 ty errors introduced by the
automated rightyper pass. Changes are annotation-only.
- PathInfo.__init__ path_info parameter widened from None to PathInfo | None
- Removed spurious self: "picopt.xxx.Y" annotations naming wrong modules
- Simplified **kwargs bogus unions to Any for variadic super().__init__ passthroughs
- config/handlers.py handler_stages/convert_chain/native types widened to proper base types
- scheduler.py _trigger_fail_fast/_cancel_subtree widened Exception to BaseException | None
- detect_format.py filtered image.info keys to str; suppressed dynamic mpinfo attr
- pil_convertible _file_format image_class widened from BmpImageFile to Image
- tar.py removed type[TarGzDetector] cls annotation violating LSP
- tests: PosixPath -> Path; ts:None/config:None -> float|None/dict|None
- Cleaned unused TYPE_CHECKING imports left behind after removing bad annotations
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* correct typing changes
* revert to python 3.10 and remove righttyper
* update devenv
* remove quoted self types
* remove self types
* format & fix
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix bun install on docker and gha compose
* remove package-lock.json
* update devenv
* only copy bun deps first for dockerfile
* update devenv
* update devenv
* Fix BufferedReader pickling error when timestamps file exists (#107)
* fix pickling of ContainerHandler when a timestamps file exists
When a prior picopt run leaves a .picopt_treestamps.yaml, loading it via
ruamel.yaml leaves the YAML instance holding a Reader with an open
BufferedReader. Submitting an UnpackJob (or RepackJob) to the
ProcessPoolExecutor then fails with "cannot pickle 'BufferedReader'
instances" because the handler carries that Grovestamps through its
_timestamps / _skipper attributes.
Add __getstate__ on ContainerHandler that nulls both fields before
pickling. The scheduler owns the real Grovestamps on the main thread;
workers never actually used it (in-archive timestamp consumption
happened in a pickled copy whose mutations never propagated back).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* remove clean_for_repack, superseded by __getstate__
The method nulled the same two fields __getstate__ now nulls at pickle
time, and its one caller always ran on a handler that had already
round-tripped through a worker — so the fields were already None.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* print exception traces
* update deps
* bump news and version
* install poppler utils in docker for pdf tests
* update deps
* format dockerfile
* add bunx support
* bump version and news to v6.2.0
* try with svgo in packages
* Squashed commit of the following:
commit 83a5253785fccc471a6dbd75b4d1eba3074c9e8c
Author: AJ Slater <aj@slater.net>
Date: Wed Apr 29 02:18:19 2026 -0700
bump version and news
commit 9b03aed2141c50e3a088bc6c66bdc097cca7ced4
Author: AJ Slater <aj@slater.net>
Date: Tue Apr 28 20:44:04 2026 -0700
update treestamp loggins
commit 1b7bdc712edc80fc7cf67dc8d7c05f169462d764
Author: AJ Slater <aj@slater.net>
Date: Tue Apr 28 20:32:59 2026 -0700
fix implementation of timestamp loggins to log after not before
commit 8d25937c813ada944d120f3add9ae7b0852d6570
Author: AJ Slater <aj@slater.net>
Date: Tue Apr 28 20:25:28 2026 -0700
Log timestamp load/dump and surface INFO at default verbose=1 (#111)
Treestamps 4 dropped its own load/dump prints, and picopt's new logger
mapped INFO to verbose>=2, so default runs went silent for messages the
old termcolor Printer always showed (config-style force_verbose=True).
- _VERBOSE_LEVEL: bump so verbose=1 (the argparse default) emits INFO,
matching the old printer's "force_verbose" tier.
- walk.py: log "Loading timestamps for: …" before Grovestamps init and
"Dumping timestamps for: …" before dumpf(). INFO renders cyan via the
existing LEVEL_STYLES.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
commit 4415376f5f263cb6dedda5657765c745ca952f6f
Author: AJ Slater <aj@slater.net>
Date: Tue Apr 28 18:41:23 2026 -0700
Pre-walk file count for a determinate progress bar (#110)
* Pre-walk file count for a determinate progress bar
Without a total the bar showed only an indeterminate spinner + count.
Add ``Walk._count_total`` that mirrors ``walk_file``'s recursion gate
(symlinks, timestamp filenames, ignore patterns, recurse flag) so each
non-recursing visit contributes one mark — matching the events the
scheduler dispatches through Reporter — and pass it as ``total=`` to
``make_progress``.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Use os.scandir for the pre-walk count
DirEntry caches ``is_dir`` / ``is_symlink`` from the directory listing,
so deep trees skip an extra ``stat`` per entry — meaningful on slow or
network filesystems.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
commit 6a46d432777367a00ede8b183dfe8ff3bbf6a725
Author: AJ Slater <aj@slater.net>
Date: Tue Apr 28 17:48:59 2026 -0700
Replace termcolor Printer with loguru + rich logger (#109)
The old Printer class wrote dots and messages directly to stdout from
both the main process and worker processes. This change discards it for
a centralized logger modeled on nudebomb's progress branch:
- New picopt/log/ package: shared rich Console, loguru sink, a streaming
CharStreamColumn progress bar, a Stats + render() summary, and a
Reporter that bundles them and dispatches each ReportStats outcome.
- All call sites converted: Printer.{saved,converted,lost,error,skip,
warn,config,...} → logger.* and progress.mark_*. Worker-side dot and
lifecycle calls dropped — workers can't reach the parent's live region,
so per-file progress is now driven from the scheduler when each result
comes back.
- Centralized MARKS table in picopt/log/styles.py drives the streaming
chars, the loguru sink colors, the summary table row colors, and the
--help epilogue legend, so the same outcome reads identically
everywhere. Style choices mirror the old termcolor palette so longtime
users see the same colors for the same outcomes.
- Scheduler now takes a Reporter; Totals removed. report.py is a pure
data class (ReportStats) with no printer dependency.
- doctor.py and the cli help epilogue rewritten with rich (rich.markup
escape() for path strings).
- Grovestamps now constructed with verbose=0 so treestamps's internal
printer doesn't bypass the rich Live region.
- loguru~=0.7 added; termcolor dropped.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
commit 03cdcb018dce4049b6e4de54e71b50df33ddfd15
Author: AJ Slater <aj@slater.net>
Date: Tue Apr 28 17:34:02 2026 -0700
move set_jpeg_xmp into jpeg plugin
commit b28337465aacbad6a65808297dc4bf25115abfd7
Author: AJ Slater <aj@slater.net>
Date: Tue Apr 28 17:06:50 2026 -0700
update devenv. treestamps 4, rich 15
* fix pyright warnings
* ty ignore
* update treestamps
* fix MPO error for 6.3.1
* Migrate to confuse 2.2.0 by replacing AttrDict with PicoptSettings (#114) (#115)
Confuse 2.2.0 made ``AttrDict`` a properly-parameterized generic; an
unparameterized ``AttrDict`` resolves to ``AttrDict[str, object]``, so
every downstream access — ``cfg.bigger``, ``int(cfg.jobs)``,
``cfg.paths`` flowing into ``Iterable[str]``, etc. — now reads as
``object`` and stops type-checking. We had ~31 basedpyright errors after
the bump, all rooted at AttrDict consumers.
Fix: keep using confuse for what it's good at — YAML / env / CLI parsing,
type coercion, requiredness — and convert the validated ``AttrDict``
into a typed frozen dataclass once. Every downstream module then takes
``PicoptSettings`` instead of ``AttrDict``.
- ``picopt/config/settings.py``: new ``PicoptSettings`` (and nested
``ComputedSettings`` / ``IgnorePatterns``) frozen dataclasses mirroring
the existing ``MappingTemplate`` schema. ``Sequence(...)`` fields are
``tuple[X, ...]`` because the dataclass is frozen. ``handler_stages``
is typed ``dict[Any, Any]`` to avoid an import cycle with
``picopt.plugins.base``; consumers know its concrete shape.
- ``picopt/config/__init__.py``: ``PicoptConfig.get_config`` now returns
``PicoptSettings``. New ``_settings_from_attrdict`` does the
field-by-field copy (typed ``Any`` so it can read AttrDict attributes
directly — the conversion *is* the schema escape hatch).
- All consumers (``picopt/path.py``, ``picopt/report.py``, ``picopt/walk/*``,
``picopt/plugins/base/{handler,container}.py``) replace
``from confuse(.templates) import AttrDict`` with
``from picopt.config.settings import PicoptSettings``.
- ``Walk._init_timestamps`` builds a shallow ``Mapping[str, Any]`` for
Grovestamps' ``program_config=`` so we don't have to serialize the
whole frozen dataclass (especially the computed sub-fields, which
carry ``re.Pattern`` and class-keyed ``dict`` objects).
- ``is_path_ignored`` switched from ``ignore and bool(...)`` to
``ignore is not None and bool(...)`` — its return type is now
honestly ``bool``, not ``bool | re.Pattern | None``.
- ``Walk._dump_timestamps`` joins the dumpf return paths via ``str(p)``
(treestamps' ``dumpf`` returns ``tuple[Path, ...]``).
pyproject pin: ``confuse~=2.1.0,<2.2.0`` → ``confuse~=2.2.0``.
Verification: ruff clean, basedpyright clean (0 errors, was 31),
``uv run pytest --deselect …test_timestamp_parents`` 149 passed
(deselected test is a pre-existing treestamps v4 regression),
``radon cc`` all A/B, CLI smoke run with env var + CLI flag still
layers correctly.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Drop dead isinstance check; split Scheduler._submit_ready_job (#116)
Two warnings flagged by `make typecheck complexity`:
- ``picopt/config/__init__.py``: confuse 2.2.0 already types
``config.get(MappingTemplate)`` as ``AttrDict[str, object]``, so the
defensive ``isinstance(ad, AttrDict)`` check after it was statically
always-true (reportUnnecessaryIsInstance). Drop it and the now-unused
``AttrDict`` import.
- ``picopt/walk/scheduler.py``: ``_submit_ready_job`` was rank C in
radon. Split into three:
* ``_drop_cancelled_ready_job`` — counter cleanup for a leaf of a
cancelled subtree (UnpackJob/RepackJob skip; only leaf jobs
decrement the parent counter).
* ``_track_submitted_job`` — record an in-flight future under the
right map and update node state.
* ``_submit_ready_job`` — orchestrate: pop, cancelled-skip, submit,
track.
``_submit_ready_job`` is now rank A; the new helpers are A/B; behavior
unchanged.
Verified: ``make typecheck`` 0/0/0; ``make complexity`` no findings;
``uv run ruff check picopt/`` clean; ``uv run pytest`` 150 passed,
6 skipped.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* update deps
* bump version and news to 6.4.0
* update devenv
* Format argparse help with rich-argparse (#118)
Switch picopt's CLI to RawDescriptionRichHelpFormatter so the help
output is colorized. A PicoptHelpFormatter subclass adds one extra
highlight regex matching every registered format string (PNG, ZIP,
WEBP, …) under the `metavar` named group, so format names mentioned
inside help text share the color of their corresponding FORMATS /
EXTRA_FORMATS / CONVERT_TO metavars.
Rewrite the dot-color-key / doctor-mode epilog as Rich markup instead
of capturing ANSI from a Console. rich-argparse renders descriptions
and epilogs through `console.use_theme(Theme(self.styles))`, so the
epilog can use [argparse.groups], [argparse.prog], and [argparse.args]
to keep its section header, program name, and `doctor` subcommand
visually consistent with how the rest of the help is rendered.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* bump version and news to 6.5.0
* typecheck
* Defer animated WebP duration extraction to handler (#119)
detect_format eagerly iterated every frame of every animated image to
populate info["durations"], which only WebPMuxAnimatedLossless ever
consumed -- and that handler already had a webpmux-based fallback
(_read_durations) for when PIL's frame iteration was unreliable.
Removing the eager extraction speeds up format detection on animated
images (animated PNG ~11x, animated GIF ~4x in local microbenchmarks)
and simplifies the webp.walk path to always use the webpmux subprocess
fallback, which is the more reliable source per the original "PIL
frequently fails to populate per-frame durations on WebP" comment.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* bump news for animated deferall
* Cache file header bytes on PathInfo for detectors (#120)
PdfDetector was opening every file and reading 1024 bytes for every
detection attempt -- including non-PDF files where it would read,
fail the magic check, and return None. On a directory of non-image
non-archive files, this was ~14us per file (88% of the non-PIL
detector chain cost).
PathInfo now lazily reads up to 4 KB of file head into _header_bytes
on first access. PdfDetector consumes path_info.header_bytes() instead
of reopening the file.
Local benchmark of running the full non-PIL detector chain on
non-image files (pyproject.toml, README.md, uv.lock, Makefile):
~16us before -> ~2us after (8x).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* bump news
* Single Image.open in _extract_image_info; read metadata before verify (#121)
PIL's verify() closes its internal fp for some Path-opened formats
(notably GIF), which then breaks lazy attrs like is_animated and
n_frames. The previous code worked around this by opening the file
twice -- once to verify, again to read format/info.
Reading the lazy attrs before verify() preserves the safety check
(verify still runs and propagates corruption errors via the same
exception path) while collapsing to a single Image.open. Confirmed
on Pillow 12.2 against GIF (still + animated), TIFF (single + multi-
frame), MPO, animated PNG, animated WebP, JPEG, BMP, PPM.
_extract_image_info per-file (warm cache, microbenchmark):
test_animated_gif.gif 165us -> 95us
test_animated_png.png 145us -> 113us
test_animated_webp.webp 66us -> 30us
test_png.png 74us -> 35us
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* bump news
* update devenv and fix typechecking
* remove use of prettier sh plugin'
gi tpush
* fix data loss when converting without external tools
pil_save short-circuited format conversion when the input format was in
the handler's accepted set, even when the target format differed. This
caused PNG→WebP via PIL fallback to return the original PNG buffer
unchanged, which the cleanup code then deleted without writing a WebP.
Also fix pipeline selection ignoring tool.required=False, which caused
PNG optimization to be skipped entirely when pngout was absent despite
pyoxipng being available. Add PIL fallback for GIF when gifsicle is
missing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* doctor: show platform-specific install hints for missing tools
Detect the package manager (brew/apt/dnf) and display the install
command next to each missing external tool in `picopt doctor` output.
Also remove obsolete mozjpeg/jpegtran system install instructions from
the README — mozjpeg is now a bundled Python dependency that installs
automatically with picopt.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* decomplexify doctor
* update deps
* bump version and news
* update devenv. remove old prettier sh plugin
* replace pyoxipng with oxipng-pybind (#124)
Swaps the deprecated pyoxipng binding for oxipng-pybind, which wraps
oxipng v10.1.1 (vs v9.x) and is actively maintained. The module import
name is unchanged (import oxipng), so the only code change needed was
fixing one deprecated call: StripChunks.safe() → StripChunks.safe (enum
value, no parentheses). All 150 tests pass.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* update deps
* bump news and version to v6.5.2
* update deps
* update deps
* Add memory-aware scheduling to prevent OOM on large archives
Optimizing large archives (e.g. multi-GB comic books) could exhaust RAM
and get the process OOM-killed. Each container holds its whole archive,
all decompressed members, the optimized copies, and the output buffer in
memory at once, replicated into each parallel worker, and nothing bounded
the in-flight total beyond a future-count cap.
Give the scheduler a memory budget: estimate each top-level archive's peak
resident memory (~3x its on-disk size) and only admit a new one while it
fits the budget, with a forward-progress guarantee so a single archive
larger than the whole budget still runs (alone). The gate scans past
budget-blocked items so in-flight work still frees the budget, avoiding
both deadlock and head-of-line blocking.
Add a --memory-limit option (e.g. 8G, 512M) that sets the budget as an
approximate peak-memory target; the default is two-thirds of detected RAM.
It is excluded from the timestamps config so it never invalidates stamps.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* update deps and format
* Fix basedpyright and ty diagnostics
Convert mypy-style `# type: ignore[arg-type]` comments in the scheduler
memory test to the project's paired `# pyright: ignore # ty: ignore`
convention (neither checker honors mypy-style comments), and make the
`_charge_info` match exhaustive with an explicit `case _` for the
intentionally-exempt RepackJob path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* update devenv
* remove pytest-gitignore
* Fix critical data-loss bugs in file replacement and archive repacking
- Write optimized results to a sibling temp file and atomically replace
the original; a crash, OOM-kill, or full disk mid-write can no longer
destroy the user's file.
- Keep members whose optimization errored (and nested containers whose
repack failed) in the repacked archive instead of silently dropping them.
- Open nested archives from their in-memory buffer; archives inside
archives previously always failed to unpack with FileNotFoundError.
- Discarded conversion results (bigger output, dry-run) no longer stat a
never-written path and are reported as skipped instead of errors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Reduce _select_pipeline_for_handler cognitive complexity
Extract per-tier tool picking into _pick_tier_tool and drop the
redundant empty-pipeline guard; behavior is unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Fix scheduler and walk error-path correctness
- A member that fails format detection (e.g. a decompression bomb) is
recorded as one error and passed through to the repacked archive
instead of aborting the entire run and losing all timestamps.
- Directories and containers with errored children no longer get
timestamps; previously the compacted dir stamp covered failed files
so they were silently skipped on every subsequent run.
- Failed top-level containers now notify their directory tracker, which
previously leaked and blocked ancestor timestamp finalization.
- Nested container repacks hydrate through the parent handler so
conversion renames (e.g. inner CBR -> CBZ) are no longer lost.
- Requested container conversions repack even when every member was
already optimized; CBR -> CBZ previously did nothing in that case.
- Ignore patterns now respect filesystem case sensitivity (the flag
was inverted), timestamps files are matched by basename, and FIFOs/
sockets are skipped instead of blocking the walk forever.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Preserve archive metadata fidelity through repack
- Archive comments (zip, rar) are read during walk and written back on
repack; _set_comment previously had no call site so comments were
always destroyed (CBZ comments commonly hold ComicBookInfo tags).
- Repack preserves original member order via a walk-time archive_index
on PathInfo; the optimized-contents set previously scrambled order,
which broke EPUB's mimetype-first requirement and reproducibility.
- Member times before 1980 (or missing) are clamped to the zip epoch
instead of crashing zipfile on conversion.
- Converted members choose compression by content: text deflates,
already-compressed image formats are stored. Previously rar/tar/7z
conversions wrote every member uncompressed, and --strip-metadata
wastefully deflated incompressible images.
- Legacy zip member names that are really UTF-8 misread as CP437 are
repaired at repack instead of being re-encoded as mojibake forever.
- Tar links survive repack: symlinks/hardlinks keep their type, target,
and zero size; link targets are no longer duplicated into link
entries (or crashes on external targets).
- 7z member mtimes are restored after repack; py7zr's writef API stamps
members with now() and offers no public override.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Fix config layering, validation, and doctor accounting
- Config-file and env values for boolean options and verbose are no
longer silently shadowed: un-passed CLI flags now default to None,
which confuse drops, so lower layers stay visible. CLI flags that ARE
passed still override everything.
- -I/--no-default-ignores without -i no longer aborts at startup (the
computed ignore patterns are validly None).
- Unparseable --memory-limit values abort with a clean config error
instead of silently falling back to auto, and unparseable --after
values no longer dump a raw dateutil traceback. Timezone-aware
--after values are interpreted correctly instead of as local time.
- picopt doctor counts requirements per pipeline tier instead of per
tool; tools within a tier are alternatives, so healthy installs no
longer report missing-required tools and exit 1.
- Lowercase format lists are accepted for -x and -c exactly as for -f.
- README no longer documents the removed PICOPT_ENABLE_IMG2WEBP
environment variable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Fix webp lossless detection, png depth parsing, and handler edge cases
- is_lossless() parses the RIFF chunk structure instead of searching
for byte strings. The old code searched the whole file via mmap for
filesystem input (metadata bytes could fake losslessness) but only
128 bytes for in-memory input (a large ICC/EXIF chunk before VP8L
misdetected lossless in-archive WebPs as lossy). It also closed the
caller's buffer and crashed mmap on empty files.
- png_bit_depth() returns None for truncated PNGs instead of a fake
0 depth.
- run_ext_fs() cleans up spilled temp files when the external tool
fails instead of leaking them.
- --preserve survives chown permission failures (non-root users) and
still restores mode and mtime instead of reporting the whole
optimization as an error.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Make PDF and SVG optimization safer
- Repacking preserves PDF encryption: owner-password-restricted PDFs
open without a password and were silently saved decrypted. qpdf
cannot decode streams while preserving encryption, so encrypted PDFs
get the weaker (but protected) rewrite.
- Optimized JPEGs are written back with their original /DecodeParms
(e.g. /ColorTransform), which pikepdf's write() otherwise drops.
- Signature detection also looks for /Type /Sig objects, catching
signed PDFs from producers that never set the AcroForm SigFlags bit.
- svgo runs with an explicit config instead of the default preset:
removeViewBox is disabled (it breaks responsive scaling), and title/
desc/metadata elements follow the --strip-metadata flag instead of
the preset's whims.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Survive symlink loops and unreadable directories
- walk_dir, the progress pre-count, and the legacy timestamp import
track visited directories by (st_dev, st_ino); symlink cycles and
duplicate links are walked once instead of recursing until the
filesystem errors (or, in the pre-count, forever).
- The legacy timestamp import skips unreadable paths with a warning
instead of crashing the whole run at startup, and matches ignore
patterns with the same case-sensitivity fix as the walk skipper.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Revive in-archive entry skipping and timestamp consumption
ContainerHandler.walk() always executes in a worker process, but
__getstate__ dropped the Grovestamps (unpicklable ruamel state) it
needs, so member entry-mtime skipping, in-archive .picopt_treestamps
consumption, and the timestamps_ignore_archive_entry_mtimes option
were all dead code.
Archive handlers now carry an ArchiveStamps slice instead: the tree's
picklable config plus its dump_dict() serialization (the same
config-headed mapping treestamps round-trips through disk). A real,
detached Treestamps is rebuilt lazily in the worker — never set() or
dumped, so it never touches the WAL — preserving full treestamps
semantics: ancestor-max lookups and config-checked loading of stamp
files found inside archives.
Members whose entry mtimes predate the archive's stamp are now
noop-copied instead of re-optimized when a stamped archive is
modified, stamp files inside archives are consumed and dropped on
repack, and -E disables entry-mtime skipping as documented.
timestamps_ignore_archive_entry_mtimes joins TIMESTAMPS_CONFIG_KEYS
since flipping it changes which members get optimized.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Improve scheduler-thread and archive throughput
- Archive member format detection (PIL open + verify per member) moves
into the unpack workers via predetect_format(), which caches onto
PathInfo; the scheduler thread previously decoded every member of
every archive serially.
- The progress pre-count no longer re-walks the whole tree when the
progress bar is disabled (-q).
- Top-level results no longer pickle their optimized bytes back to the
main process just to be discarded; only in-container entries return
data for their parent's repack.
- Items deferred by the memory gate wait in a FIFO gated queue retried
head-first per tick instead of rescanning the entire ready queue on
every completion (O(queue) per event under memory pressure).
- 7z archives extract all members in one pass; solid archives
previously decompressed from the start for every single-target
extract, O(n^2) over members. The extraction cache is dropped before
the handler pickles back to the main process.
- Add bin/benchmark.py: wall-clock + per-run peak RSS harness over
synthetic fixture corpora, for measuring future changes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Extract dir timestamping, wire staging cleanup, untangle reporter dispatch
- The per-directory completion tracking and compacted timestamp writes
move from the scheduler into walk/dir_timestamps.DirTimestamper; the
scheduler delegates its begin/seal/cancel API.
- All five container pending-counter decrements route through one
_child_done() method; _maybe_start_repack's state guards make it safe
on every completion path.
- The staging-dir cleanup design is finally wired: handlers expose
get_staging_dir(), the scheduler stores it on the node after unpack,
and rollback/fail-fast/final cleanup actually remove worker tmp dirs.
Animated-WebP frame directories previously leaked on cancellation.
- Reporter outcome dispatch is an explicit match instead of
getattr(f"record_{kind}") string dispatch that failed silently on
renames.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Split the webp plugin into a package
plugins/webp.py (759 lines, five handlers, four pack strategies) becomes
plugins/webp/: const, tools, static, and animated modules with the
PLUGIN descriptor in __init__. Plugin discovery now accepts subpackages.
The CWebPTool.probe() -> IS_MODERN_CWEBP class-attribute side effect is
replaced by an is_modern flag on the shared, probed-once CWEBP_TOOL
singleton that WebPLossless's PIPELINE holds and both consuming handlers
read at construction time. The tools no longer import handler classes at
all — they duck-type on the args-provider methods — removing the
tool/handler import cycle the isinstance checks required.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Deduplicate handler routing and document the PathInfo contract
- plugins.pick_route_handler is the one routing decision: the factory
uses it at runtime and config/handlers uses it to build the startup
formats summary, replacing an 8-parameter re-derivation whose logic
had already drifted from the factory's (it ignored empty-pipeline
availability). The log can no longer disagree with the walk.
- PathInfo's class docstring documents the mutable-identity contract:
instances cross the process boundary and are mutated by the scheduler
after hydration, with cache invalidation owned by rename()/set_data().
- computed.handler_stages is structurally typed dict[type, tuple];
the concrete Handler/Tool annotation is impossible because
plugins.base imports settings and basedpyright's reportImportCycles
counts even TYPE_CHECKING edges.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Memoize tool probes
probe() caches its ToolStatus per instance and concrete probes move to
_probe(). Tools are module singletons, so each external tool spawns its
--version subprocess once per process — a prerequisite for per-directory
config files, whose confuse rebuilds rerun the pipeline probe loop.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Add per-directory .picopt.yaml config files and config-writing flags
Ported from nudebomb's implementation (same confuse + treestamps stack).
A .picopt.yaml placed in a target directory applies to that directory
and everything beneath it. Files are discovered from each processed
file's directory up to the CLI target root (never above it); deeper
directories win over shallower ones, and env vars and CLI options still
win over every directory file — implemented by rebuilding a fresh
confuse Configuration per unique directory chain through the extracted
_build_config layering core. DirConfig caches resolutions per directory
with an identity fast path when no config applies, and isolates
malformed files (logged once, recorded in stats, fall back to run-wide
settings). Run-mode keys (dry_run, list_only, timestamps) are pinned to
run-level values: a per-directory dry run with run-level timestamps
would stamp files that were never optimized.
Per-directory settings reach everything: guards (ignore/symlinks/
recurse) via per-directory skippers and the pre-count mirror, handlers
via the factory's settings parameter (formats/convert_to/bigger/
keep_metadata/... travel into workers), archive members by inheriting
their container's config (nesting inherits automatically), and repacks
run under the handler's own config. .picopt.yaml files are skipped as
optimization targets by basename.
With -t, a sha256 fingerprint of every .picopt.yaml under the targets
joins the treestamps program-config check, so editing, adding, or
removing a directory config re-processes its tree (over-invalidation,
never a wrong-skip).
New write flags, all validate-before-persist with round-trip YAML
merges that preserve comments and unrelated keys, written owner-only:
-w/--write-config (user config), -W/--write-dir-config (a .picopt.yaml
in each target directory), and --write-config-file PATH. Run-mode
flags and target paths are never persisted.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Add fail-fast, cancellation, idempotency, and pixel-identity tests
- --fail-fast cancels every live top-level container and clears both
scheduler queues on the first repack error; --fail-fast-container
escalates an inner failure to its subtree root while leaving sibling
top-level containers untouched; a leaf result arriving after its
parent's cancellation is dropped without touching the parent's
contents.
- Optimization converges: a second pass may only shrink files (the
multi-tool png pipeline legitimately shaves another byte when one
tool's output enables another) and every file reaches its fixed
point by the third pass.
- Lossless paths are pixel-exact: png optimization, png-to-webp, and
gif-to-png conversions decode to identical RGBA pixels.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Update NEWS for v6.6.0
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MgZ7kxNgvEWuSEKeTJMJcH
* Add missing v6.6.0 news items
Fill in five user-facing fixes that landed since v6.5.2 but were absent
from the v6.6.0 entry: archive conversions recompress members by content,
--preserve survives non-root chown failures, temp files and animated-WebP
frame dirs are cleaned up on failure/cancellation, timezone-aware --after
is honored, and -I without -i no longer aborts at startup.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* version 6.6.0
* Fix repeated "Optimizing formats" banner with -W and planted .picopt.yaml
The verbose "Optimizing formats" banner is a run-wide summary, but it was
emitted from set_format_handler_map(), which _build_config() re-runs for
every per-directory .picopt.yaml resolution during the walk. With -W (or
any planted directory config in the tree root), every directory finds a
config in its ancestry, skips the "reuse run-wide settings" fast path, and
rebuilds the config — reprinting the banner once per directory.
Gate the banner behind a print_summary flag so only the top-level
get_config() build emits it. Move the summary computation into
_log_formats_summary() so the per-directory rebuilds skip it entirely.
Add a regression test asserting the banner prints exactly once under -W.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* bump version to 6.6.1
* bump news
* Fix animated WebP optimization failing on zero-duration frames
img2webp rejects any frame duration <= 0 ("Invalid negative duration"),
aborting the whole animated-WebP repack. Some sources (e.g. scraped
animated WebPs) carry 0ms frame durations, so clamp each duration up to
img2webp's 1ms minimum in Img2WebPAnimatedLossless.img2webp_args.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* update deps
* Clear basedpyright typecheck warnings
- Use a real PicoptSettings (not SimpleNamespace) in the img2webp
duration test so config assignment type-checks.
- Mark the module-level pytest autouse fixtures with
`# pyright: ignore[reportUnusedFunction]`; they are used by pytest,
not referenced by name.
- Disable reportImplicitStringConcatenation: ruff format canonicalizes
adjacent string literals to implicit concatenation, so the check can
never be satisfied.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix news formattign
* Reduce cyclomatic complexity of four rank-C functions
radon cc flagged four functions at rank C. Extract behavior-preserving
helpers to bring each to rank A/B:
- WalkSkipper._is_skippable (16): split into _skip_reason and the
disk-only _skip_reason_fs.
- Walk._count (12): extract the recursion-stop predicate _count_stops_here.
- pick_route_handler (12): fold the convert gate into _pick_convert_handler
and extract _can_pack_repack for the repack check.
- Handler._cleanup_after_optimize (11): extract the _should_replace predicate.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Print verbose config summaries only once, not per directory config
Per-directory .picopt.yaml resolution rebuilds the confuse config for
every directory in the tree, repeating the timestamps, ignore, after,
and memory-limit log lines each time. Gate them on the existing
print_summary flag, as set_format_handler_map already does.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* update deps and bump version to 6.6.2
* Add v6.6.2 news entry for repeated verbose summaries fix
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix timestamps silently invalidated by tree-root .picopt.yaml configs
The treestamps program config was recorded from the run-level CLI
config, so identical options expressed via a tree root's .picopt.yaml
instead of flags always mismatched and every stamp was silently
discarded, re-optimizing whole trees.
Each tree now records its own effective config: the new Grove builds
one Treestamps per top path from DirConfig.get_tree_settings, which
resolves the root's .picopt.yaml beneath CLI and env. timestamps: true
at a tree root now enables stamping without -t (dry run and list still
disable it, and subdirectory configs still can't toggle it mid-walk).
The directory-config fingerprint is now per tree and value-based: it
hashes the parsed picopt section of each config strictly below the
root, so comment and formatting edits, -W rewrites, running with a
different set of top paths, and file-vs-directory targets of the same
tree no longer invalidate stamps — only real option changes do. The
root's own config is excluded because its options are recorded as
values.
Stamps discarded on a config mismatch now warn with the differing keys
(treestamps 4.1.1); stdlib logging is intercepted into loguru so the
warning renders through the shared console, and log messages escape
Rich markup so bracketed names like "Title [en]" display intact.
Migration: existing stamp files mismatch once after this upgrade (now
loudly); a single -N run preserves old stamps instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T7R5s6frL7jEykysAzjttq
* update deps
* Report corrupt images as clean warnings instead of dumping tracebacks
PIL raises a plain OSError when it recognizes an image format but the
contents are corrupt (e.g. a BMP with pixel depth 0), which escaped the
detection probe and printed a full stack trace. Detection now wraps it
in UnreadableImageError, reported as a one-line warning that is also
counted and listed in the summary; the file is skipped untouched.
The per-file exception guards now skip the stack trace for any
PicoptError — extending the contract cli.main already had: expected
conditions print their message, only unexpected exceptions dump stacks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T7R5s6frL7jEykysAzjttq
* version 6.6.3
* update deps
* Fix ty and remark-lint warnings
- Drop the `cls: type[Any]` annotation on the RarDetector/ZipDetector
`identify` classmethods; ty flagged the widened first parameter as an
invalid override of `Detector.identify`.
- Replace remark's `no-duplicate-headings` with
`no-duplicate-headings-in-section`. A changelog necessarily repeats
`### Fixes` under each version; the in-section rule still catches real
duplicates.
Also includes the in-progress dependency bumps and ruff rule additions that
were already in the working tree.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Jxl format (#129)
* Add JPEG XL support
Read and write JPEG XL via the pillow-jxl-plugin codec, as a new
auto-discovered plugin. JXL is optimized by default and is now the
preferred still conversion target ahead of WEBP and PNG.
Only lossless JXL is re-encoded; lossy (XYB encoded) JXL is left alone,
the same policy lossy WebP already has. The codec exposes no such signal,
so picopt/pillow/jxl.py parses the codestream image header to read
xyb_encoded, and refuses files carrying JPEG reconstruction data, whose
re-encoding would silently destroy the ability to restore the source JPEG.
New --convert-jpeg-to-jxl converts JPEG to JXL by lossless bitstream
reconstruction, which is reversible byte for byte. It requires -c JXL as
well: picopt otherwise never converts from a lossy source, so -c JXL alone
must not sweep up a whole photo library. The codec only takes its
reconstruction path when PIL knows the source filename, which it does only
for path opens, so the tool opens by path and spills to a temp file for
inputs that have none. That makes it work for archive members too.
Three smaller things the codec needed:
- It registers no save_all handler, and pil_save always passes save_all,
which raised KeyError. The single-frame handler is registered for it;
animated JXL is unsupported by the codec regardless.
- Given no exif kwarg it reads EXIF off the opened image, which quietly
defeated --strip-metadata. prepare_info now passes an empty exif.
- It rejects palette and other modes, so ImageHandler gained a
prepare_image hook, symmetric with prepare_info, to substitute one.
Handlers can now declare CONFIG_ENABLED_KEY, a config flag that must be
true for them to be probed at all. Failing it leaves the handler out of
the probed handler map, which routing already reads as unavailable, so the
startup conversion banner stays accurate for free.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Gate WebP to JXL conversion behind --convert-webp-to-jxl
JPEG XL support in browsers and image tooling is still thin, so converting
away from WebP, which every browser reads, should be a deliberate act
rather than a side effect of asking for -c JXL.
JPEG and WEBP are both formats picopt processes by default, which means
the usual "name the source with -x" gate cannot express a choice about
them: -x JPEG and -x WEBP are no-ops. Each therefore carries its own flag,
and a plain -c JXL once again converts only GIF and PNG, as the README has
always described. The previous commit had quietly made WEBP a third
convert-by-default source.
The flag hangs off the handler class, and JxlLossless also serves PNG, GIF
and native JXL, which must stay ungated. So the WebP route now points at a
JxlFromWebP subclass that exists only to carry CONFIG_ENABLED_KEY.
Converting out of JXL into a more widely readable format stays ungated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* update deps
* update deps
* update deps
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* update deps
* v6.8.0: timestamps survive upgrades that record new options
Adding an option to the recorded timestamps config used to invalidate
every existing timestamp file, re-optimizing whole trees. Picopt now
passes its packaged defaults to treestamps 5.0.0, which fills keys
missing from either side before comparing, so only a changed option
value invalidates. Timestamp files written before 6.7.0 added the JXL
options, and before 6.6.2 added the directory config fingerprint, are
honored again.
Timestamp files now open with a comment saying what they are, that they
are machine-written, and that deleting one re-optimizes that tree. The
redundant treestamps_config block is gone; older files still load.
Discarding timestamps because a sub-directory .picopt.yaml changed now
names it that way instead of printing the internal fingerprint key.
Replace the local directory config fingerprint with the identical
treestamps implementation, pinned by golden digests upstream.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* update deps
* Write config files atomically
merge_config_file truncated the target before writing, so a crash, a
full disk, or a kill during -w, -W, or --write-config-file could
destroy a hand-authored .picopt.yaml or user config. Write to a unique
sibling temp file, fsync, then replace, matching the discipline the
image handler and treestamps already use for files that matter less.
The temp name is unique rather than a fixed sibling so two concurrent
writers cannot unlink each other's temp mid-write, and the owner-only
mode is set before the rename so the target is never briefly wider.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Note the atomic config writes in the v6.8.0 news
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* v6.8.1: rewrite timestamps discarded for a config mismatch
Requires treestamps 5.0.1. A run that discarded a timestamps file for a
config mismatch only rewrote it if it also optimized something. A run
that optimized nothing left the rejected file on disk, so it warned
about the same mismatch every time.
uv.lock is left at treestamps 5.0.0 until 5.0.1 publishes to PyPI.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* update deps
* update devenv
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lossless JXL format support