perf(extract): remux all-CCITT sources instead of decoding, finer chunks#35
Merged
Conversation
Nothing in the pipeline measured where a run's time went: ProgressEvents
carry no timestamps and the stage logs no durations, so optimization
work had no baseline to argue against. This adds the measurement layer:
- --timings: a StageTimingSink (composed in the CLI shell) prints a
stable, machine-parseable per-stage breakdown to stderr when a run
ends ("timing: <stage> = <seconds>s (<percent>%)"), including the
still-open stage on failure.
- PipelineRunner logs each stage directory's byte total, making the
intermediate I/O of every stage visible.
- benchPipeline: a Gradle task driving the installDist launcher with
--timings (PipelineBenchmark, test sources), measuring E2E wall, the
per-stage medians, peak RSS via /proc VmHWM, and output size, over a
-Pjobs sweep; writes pipeline/docs/perf-baseline.md.
- createSampleScan: a deterministic synthetic 600-dpi A5 scan book
(specks for despeckle, ±0.5° skew for deskew) so the benchmark needs
no copyrighted input and stays comparable across machines.
Baseline on the 200-page fixture (8 CPUs): conv 14.48s at -j8 —
despeckle 68%, register 22.6%, extract 7.9%, spread 1.5% — and a
3.44x scale-up from -j1, recorded in pipeline/docs/perf-baseline.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pdfimages -tiff decodes every embedded G4 image into an uncompressed TIFF (~2.2 MB per 600-dpi page; ~434 MB of transient intermediates for a 200-page book) even though the typical self-scanned source is CCITT G4 end to end. (The originally planned `-tiffcompression g4` flag does not exist on pdfimages — it is a pdftoppm option.) The extractor now picks its mode from one pdfimages -list pass: when every embedded image is 1-bpp CCITT, each chunk dumps the raw G4 streams (-ccitt) and CcittTiffs wraps them verbatim into single-strip CCITT-G4 TIFFs — a pure remux: no decode/re-encode, intermediates drop ~60x, and the image's true ppi is stamped instead of pdfimages' default 72 dpi. Because PDF's EncodedByteAlign never reaches the dumped .params file, every wrapped page is decoded back once through Leptonica as verification; a chunk that deviates in any way (params shape, count, or a wrap that fails to decode) is re-extracted decoded, which is also the whole-run mode for any non-CCITT source. The photometric mapping (-B -> WhiteIsZero, -W -> BlackIsZero) is pinned empirically by a pixel-identical round trip test. Extraction chunks also shrink from total/jobs to ~12 pages (capped at 4*jobs): fast finishers free their pool slot early, and a future streaming source can consume pages chunk by chunk. Benchmark (200-page fixture, warm median of 3, vs the PR #28 baseline): extract 1.15s -> 0.46s at -j8 (4.57s -> 0.88s at -j1), conv 49.85s -> 45.98s (-7.8%) at -j1, intermediates ~434 MB -> ~7 MB. Output validated with qpdf --check (100 spreads, linearized, no errors). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The CI spell check flags PDFBox's COSName.DECODE_PARMS (the PDF spec's own key name, which the remux test must name verbatim) and a hyphenated coinage in the extractor's javadoc. Allowlist the spec identifier — the same precedent as the veraPDF en-GB names — and use plain words. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Squash merges orphan the stack's ancestry, so the benchmark documents (regenerated on every bench run) collide as add/add between this branch and main. Align them to main's version; the round's closing PR commits the final regenerated baselines, so no information is lost from the final state. The measured numbers this PR contributed remain in its commit message and PR description. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
pdfimages -tiffdecodes every embedded G4 image into an uncompressed TIFF (~2.2 MB per 600-dpi page — ~434 MB of transient intermediates for a 200-page book), even though the typical self-scanned source is CCITT G4 end to end. The originally planned-tiffcompression g4flag turned out not to exist onpdfimages(it's apdftoppmoption), which forced a better design.What
CCITT remux mode. One
pdfimages -listpass picks the extractor's mode: when every embedded image is 1-bpp CCITT, each chunk dumps the raw embedded G4 streams (-ccitt) andCcittTiffswraps them verbatim into single-strip CCITT-G4 TIFFs — a pure remux:pdfimages' default 72 dpi.Defense in depth. PDF's
EncodedByteAlignnever reaches the dumped.paramsfile (verified against poppler 24.02 source), so every wrapped page is decoded back once through Leptonica; a chunk that deviates in any way (params shape, dump count, or a wrap that fails to decode to the listed dimensions) is deleted and re-extracted decoded (-tiff) — which is also the whole-run mode for any non-CCITT source. The photometric mapping (-B→ WhiteIsZero,-W→ BlackIsZero) is pinned empirically by a pixel-identical round-trip test (the first mapping attempt was inverted; the test caught it).Finer chunks. Extraction chunks shrink from
total/jobsto ~12 pages (capped at 4×jobs): fast finishers free their pool slot early, and a future streaming source can consume pages chunk by chunk.Measurements (200-page fixture, warm median of 3, vs #28 baseline)
Meets the acceptance rule on both prongs: ≥5% wall at
-j1and an explicit disk win.Verification
./gradlew checkgreen (all modules; ArchUnit, Error Prone, NullAway)..tifonly, stamped 200 ppi, no.ccitt/.paramsresidue, non-inverted ink).PipelineFlowTeste2e exercises the new path through despeckle → register → spread.qpdf --checkclean, 100 spreads, linearized.🤖 Generated with Claude Code