backend: the permissive runtime boundary -- convert with PyMuPDF absent - #3
Closed
ebt55 wants to merge 1 commit into
Closed
backend: the permissive runtime boundary -- convert with PyMuPDF absent#3ebt55 wants to merge 1 commit into
ebt55 wants to merge 1 commit into
Conversation
The roadmap called the licence flip mechanical: a dependency change and a default.
The built wheel disagreed. `fitz` was on the default runtime path in five stages
*past* the parser, so a wheel installed without PyMuPDF failed while importing the
writer, before any backend selection could happen.
docxout module scope `import fitz` -> gone
docxout table fitting MuPDF base-14 shaping -> Para.src_widths
docxout figures MuPDF pixmap of the clip -> Backend.render_clip
refine measurement get_text("dict") on both sides -> Backend.page_lines
verify rasterisation MuPDF pixmap samples -> Backend.render_page
ladder MuPDF base-14 shaping -> metrics.TextMetrics
tests/test_no_pymupdf.py is the proof, and it does not trust the code to avoid an
import: it installs a sys.meta_path finder that makes `fitz` *unimportable*, evicts
anything already loaded, and then converts a fixture per capability -- text-only,
tables, inline image, vector figure clip, multi-page with refinement, multi-column,
cover band, gdocs static profile -- and runs the closed loop through the permissive
path. That is stricter than a clean virtualenv, which cannot catch an import some
other module has already performed. All pass.
The writer's half cost nothing: both lanes re-measured, and not one of 224 values
moved (2 lanes x 16 documents x 7 gated metrics, compared exactly, not within
tolerance). Replacing base-14 shaping with the source's own recorded line widths is
not an approximation of the old answer -- for "is this column too narrow for
content that occupied one line in the source", it is a fact where there was a
prediction.
The refine half cost two documents, and the gate written last commit is what said
so: within-2pt 0.46 -> 0.31 on 03_tech_report_code and 0.60 -> 0.32 on
r1_reportlab_report. Cause is D2 in a second location. The loop measures
rendered_box_top - source_box_top over two differently-typeset documents; on the
core-14 source PDFium substitutes a generic ascent (0.905x size) for the real one
MuPDF reads from its base-14 table (1.075x), while on the Liberation render both
read the embedded metrics and agree. The subtraction keeps a ~0.17 x size bias,
about 1.7pt at 10pt, against a 2pt threshold.
Anchoring the loop on baselines cancels that perfectly and is measurably WRONG: the
incumbent's mean within-2pt goes 0.511 -> 0.478. It fixes 04_exec_brief
(0.22 -> 0.44) and breaks 05_memo (0.64 -> 0.48) and r1_reportlab_report
(0.60 -> 0.32) -- the same outcome as the line-box escalation already closed out in
D2, because the space_before chain the offsets feed is itself calibrated on box
tops. `refine.ANCHOR` keeps the switch and the measurement beside it so nobody
rediscovers this a third time.
So the two documents are ratified into parity_policy.json under D2, with floors,
joining the two already there. All four are core-14; every document that embeds its
fonts is untouched, which is the shape of the cause. **The accepted set grew from
two to four because the comparison got more honest, not because the backend got
worse**: until now refine.py read its measurement through PyMuPDF whichever backend
had parsed, so the candidate lane was pdfium parsing with MuPDF measuring -- a
configuration nobody could install. Parity is 0 regressions, 10 same, 2 expected
divergences, 4 accepted, and the end-to-end pdfium mean within-2pt is 0.4431 where
the hybrid reported 0.461. The honest number is the lower one.
Also here, both found by their own noise rather than by review:
* PDFium native handles were never closed, in `parse_pdfium.parse_pdf` and in
all three render paths. A parity run ended with pypdfium2 printing "The
following objects are still open and will now be closed" for 16 documents, 18
pages and 9 text pages. Interpreter exit collected them, which is not a
resource policy -- a worker converting a queue would hold a native document per
job until it died. Documents, pages and text pages now close in reverse order
of acquisition, and the warning is gone across 32 parses. Golden IR still 7/7,
so nothing was being read lazily off a handle that is now closed.
* Every LibreOffice invocation shared one profile -- a fixed temp path, for every
conversion in every process on the machine. Two concurrent conversions then
contend for it and one exits 0 with no output, which is the exact failure that
motivated using a dedicated profile at all. Default is now per-process.
* evidence.merge skipped nothing, so the final `evidence.py --out` step passed the
empty template's `parity: None` over the verdict just recorded, and a fully
green run ended with an artifact that had forgotten its own parity result. None
sections are now skipped, with a test.
* The writer's target mode was a module global that write_docx set and restored,
so two concurrent conversions with different targets could each observe the
other's line-height encoding. It travels in a frozen WriteCtx now.
* Changing the API default from 0 refine rounds to the product profile silently
gave edge_cases.py and exp_sweep.py three rounds and an oracle dependency they
did not want. Both now name RAW explicitly.
Superscript stays hardcoded False in the pdfium backend on purpose, and that is
measured rather than deferred: backend_superscript.py shows the writer never sees
the parser's flag.
What remains for the relicence is genuinely mechanical -- pypdfium2 as the runtime
dependency, pymupdf to `[mupdf]`, goldens re-frozen from pdfium, every gate number
re-recorded because the default parser changes -- plus one thing that is not an
engineering decision and is deliberately left alone here: a licensing review of the
Apache-2.0 distribution and the `[mupdf]` extra's wording.
Full CI-equivalent sequence green on the canonical Linux environment.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Owner
Author
|
Superseded by #4. PR #4 has been retargeted onto Closing as a draft slice that has been folded into the combined |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft — review work, not ready to merge. Stack 2 of 3, based on PR 1.
Makes every code path able to run without PyMuPDF. This is what the phrase
"mechanical licence flip" was hiding.
Why
The roadmap described the flip as a dependency change and a default. The built
wheel disagreed:
fitzwas on the default runtime path in five stages past theparser, so a wheel installed without PyMuPDF failed while importing the writer,
before any backend selection could happen.
docxoutmodule scopeimport fitzdocxouttable fittingPara.src_widths— the widthinferalready recordeddocxoutfiguresBackend.render_cliprefinemeasurementget_text("dict")both sidesBackend.page_linesverifyrasterisationBackend.render_pageladderTextMetricsseamProof
tests/test_no_pymupdf.pydoes not trust the code to avoid an import — it installsa
sys.meta_pathfinder that makesfitzunimportable, evicts anything alreadyloaded, then converts a fixture per capability (text-only, tables, inline image,
vector figure clip, multi-page with refinement, multi-column, cover band, Google
Docs static profile) and runs the closed loop through the permissive path. That is
stricter than a clean virtualenv, which cannot catch an import some other module
already performed.
This does not mean the shipped artifact is permissive.
pymupdfis stillPRODUCT.backendand still a hard runtime dependency. What changed is that theflip is now a dependency-and-default change rather than a rewrite.
Cost
The writer's half cost nothing: both lanes re-measured, and not one of 224
values moved (2 lanes × 16 documents × 7 gated metrics, compared exactly, not
within tolerance).
The refine half cost two documents, and the gate from PR 1 is what said so —
within-2pt 0.46 → 0.31 and 0.60 → 0.32. Cause is D2 in a second location: the loop
subtracts a source y from a rendered y over two differently-typeset documents, and
a line-box top carries a per-font metric convention that does not cancel.
Anchoring on baselines is the obvious fix and is measurably wrong: it costs the
incumbent mean within-2pt 0.511 → 0.478, because the
space_beforechain theoffsets feed is calibrated on box tops. Same outcome as the line-box escalation
already closed out in D2.
refine.ANCHORkeeps the switch and the number beside it.Also here
pages and 9 text pages still open at exit.
concurrent conversions reproduced the exact failure the profile existed to
prevent.
different targets could observe each other's line-height encoding.
GitHub Actions has not yet run these commits.
🤖 Generated with Claude Code