Skip to content

Built-in engines W28: RTF writer, OpenDocument templates and flat files, DjVu, PDF/A-2b without Ghostscript - #1719

Merged
KnifeLemon merged 10 commits into
builtin-enginesfrom
p4/w28
Nov 20, 2026
Merged

KnifeLemon merged 10 commits into
builtin-enginesfrom
p4/w28

Conversation

@KnifeLemon

@KnifeLemon KnifeLemon commented Nov 20, 2026 •

Copy link
Copy Markdown
Owner

Wave 28 of the built-in engines: an RTF writer, OpenDocument templates and flat files, DjVu, and PDF/A-2b without Ghostscript.

RTF writer

  • New rtf-writer engine (Hwp/Hwpx/RtfWriter*.cs) writing RTF from the shared document model: character and paragraph formatting, fonts with \falt, colours, styles, lists, tabs, borders and shading, tables (merges, nested tables), pictures (PNG/JPEG as is, EMF/WMF as metafiles, others as PNG), text boxes and shapes with \shprslt text, headers/footers, sections, links, fields, footnotes and endnotes; equations as text.
  • ASCII output with \uN and code-page fallback bytes under \ucN.
  • Every built-in reader's format → RTF at cost 8, ahead of LibreOffice (25) and Pandoc.
  • Limits through DocumentLimits (output size, table cells), cancellation, no file left on failure.
  • RTF reader: {\*\falt} alternates, \deftab (default 720), \tx, more list number formats, \clmrg after \clvmrg; headers/footers were always read as odd-page ones (fixed).
  • ODT reader: default tab interval from the default paragraph style (else 1.25 cm), relative to the indent or the margin per TabsRelativeToIndent.

OpenDocument templates and flat files

  • New odf-variants engine: lossless ODT↔OTT↔FODT, ODS↔OTS↔FODS, ODP↔OTP↔FODP, ODG↔OTG↔FODG (24 edges).
    • OdfVariants: template ↔ package swaps the media type in mimetype and the manifest.
    • OdfFlat: flat ↔ package, with base64 pictures ↔ Pictures/ parts (duplicates shared), embedded objects ↔ Object N/, clashing automatic style names renamed.
  • FODT/OTT/OTS/FODS are now read by the document, sheet, table and renderer paths; OTG by the drawing engine.
  • New writable format otg; fods, fodp, fodg writable. Settings schema v111: the vector donut learns .otg.
  • Flat files capped at 256 MB and read through SafeXml; pictures and objects bounded.

DjVu (#346)

  • New djvu engine (src/Filee.Engines/Djvu/): ZP coder, BZZ, JB2 with shared dictionaries, IW44 backgrounds and foregrounds, palettes, hidden text. Single-page and bundled files; indirect files refused with error.djvu_indirect.
  • PDF: a page per DjVu page at its DPI size, background JPEG plus 1-bit masks per colour, /Rotate, hidden text as an invisible searchable layer (reusing the OCR text layer). PNG, JPG and TIFF pictures of the pages.
  • Limits on pages, chunks, unpacked bytes, pixels, shapes, slices and text zones (error.document_too_complex); iterative; fuzz target.
  • Settings schema v112: the documents donut learns .djvu and .djv.
  • Checked on 22 public DjVu files (djvu.js test files, with their reference pictures): all convert to PDF; the 12 pages with reference pictures render the same (mean difference 0–1.1 of 255). Not added to the repo.
  • Some files (the DjVu v3 specification) give every word its line's box; such lines now get their words spread by length, so the text extracts in reading order.
  • PDFs are 5–19× the DjVu size: the 1-bit masks are Flate-packed. Smaller masks are a follow-up.

PDF/A-2b without Ghostscript

  • PdfAConformance: existing PDF → PDF/A-2b with PDFsharp, removing what PDF/A-2 forbids (encryption, LZW, JavaScript and other actions, embedded files, XFA, forbidden annotations and keys), print flags on annotations, one sRGB (or matching existing) output intent.
  • Refuses with pdfa.reason.* (font not embedded, CMYK without a profile, annotation without appearance, …) and leaves the file untouched; the next archiver (Ghostscript) gets it.
  • PdfAMetadata: an incremental update after PDFsharp's save writes matching Info and XMP (pdfaid part 2, conformance B), keeping /ID.
  • SrgbProfile: a 2.3 KB v2.1 ICC profile built in code from IEC 61966-2-1's numbers.
  • JobQueue tries archivers in engine order (PDFsharp before Ghostscript); the Filee renderer writes PDF/A-2b directly when the preset asks for it. PDF/A presets on PDFs work without Ghostscript.
  • Not validated with veraPDF (not installed); structural tests of the implemented rules only.

Merge notes

  • Settings schema 112 (v110 unused, v113 not needed); v111 and v112 migrations kept.
  • FormatFields, README rows, the General grade count (107) and docs/ENGINES.md rows combined across branches.

Tests

Run locally on the merged branch:

  • Release build
  • Filtered engine tests (RTF, tabs, ODF variants, DjVu, PDF/A, page writer, routes, drawings, renderer, ODT, OCR text layer, fuzz): 410 passed, 18 skipped, 0 failed
  • Core: 1793 passed, 11 skipped
  • App localization, PDF/A, engines, settings and left-out tests: 322 passed, 11 skipped, 1 failed. The failure is Engines_page_shows_versions_descriptions_and_download_speed: this PC has Tesseract installed in Program Files, which reports no version. Not related to this wave.
  • dotnet format --verify-no-changes --severity warn clean on the touched projects

🤖 Generated with Claude Code

KnifeLemon and others added 9 commits November 20, 2026 18:04
A new built-in engine, odf-variants (OdfVariantConverter), turns each
form of an OpenDocument type into the others without reading the
document into a model: ODT/OTT/FODT, ODS/OTS/FODS, ODP/OTP/FODP and
ODG/OTG/FODG. A template is the package with another media type in
mimetype and the manifest; a flat file is the package's parts put
together into one office:document (pictures as office:binary-data,
embedded objects as office:document inside draw:object) or taken apart
again (Odf/OdfFlat). Automatic styles of styles.xml that clash with
content.xml are renamed with their references.

The readers read every form: OTT and OTS directly, FODT and FODS as the
package they split into, so FODT/OTT reach PDF through the Filee
renderer and the writers, FODS/OTS reach PDF through sheet printing and
XLSX/CSV/TSV. Out, DOCX -> ODT -> OTT/FODT and XLSX -> ODS -> OTS/FODS
chain through the new engine; the ODP writer writes OTP and FODP itself
(its edges are direct only). LibreOffice keeps its edges at fallback
cost.

OTG is a new vector format; FODS, FODP and FODG become writable.
Settings v111 teaches existing vector donuts .otg. New engine texts in
all locales; README and docs/ENGINES.md drop the LibreOffice mark for
these formats.

Flat files are read through SafeXml within the document limits and at
most four times a package part; pictures decode within the part,
package and entry limits; objects are bounded in count and depth; a
flat file made from a package is bounded in size. Hostile tests build
their inputs in code; a fuzz test mutates flat texts, flat sheets and
packages turned flat.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- RtfWriter writes RTF from the document model: fonts with \falt, colours,
  styles, lists, tables with merged, shaded and bordered cells and nested
  tables, pictures (PNG/JPEG kept, EMF/WMF as metafiles, others as PNG),
  text boxes and shapes, headers and footers, sections, links, fields,
  bookmarks and notes; text beyond ASCII as \uN with a code-page fallback
  and \ucN. Output size, tables, pictures and nesting stay within
  DocumentLimits.
- RtfWriterConverter (rtf-writer) takes every built-in reader's format to
  RTF at cost 8, ahead of Pandoc and LibreOffice.
- RTF reader: font alternates (\falt), \deftab (720 twips by default),
  tab stops (\tx with kind and leader), more list number formats, \header
  and \footer as every page, \clmrg after a \clvmrg cell, and the space
  after a note's number.
- ODT reader: default tab interval from the default paragraph style
  (LibreOffice's 1.25 cm otherwise), from the indent unless settings.xml
  turns TabsRelativeToIndent off; the layout honours it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PDFs of a PDF/A preset are made PDF/A-2b by the PDF engine first, with
Ghostscript as the fallback for what it can't make conform:

- PdfAConformance changes a PDF opened with PDFsharp to keep the
  ISO 19005-2 level B rules: one GTS_PDFA1 output intent (sRGB, or a
  CMYK/RGB PDF/A intent the file brings), no encryption, LZW or external
  streams, no forbidden actions, JavaScript, AA, embedded files or XFA,
  annotations that print with only their normal appearance, cleaned
  graphics states, pictures and forms, named optional content configs.
  It refuses with a pdfa.reason.* key when a font isn't embedded, CMYK
  has no CMYK intent, an annotation has no appearance, or the file holds
  PostScript, XFA-only forms or data it can't read.
- PdfAMetadata saves it and appends an incremental update that rewrites
  the document information and XMP (pdfaid 2/B) so both match: PDFsharp
  always writes an XMP packet of its own on save.
- SrgbProfile builds a version 2.1 sRGB ICC profile from IEC 61966-2-1
  in code, so no third-party profile is shipped.
- PdfSharpConverter is an IPdfArchiver; JobQueue tries archivers in
  engine order, a refusal hands the PDF to the next, and a PDF none
  archives gets the notice with the refusal's reason.
- The Filee renderer (documents, sheets, slides, drawings) writes PDF/A
  directly when the preset asks for it.
- A PDF/A preset for PDFs no longer needs Ghostscript to be usable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Filee's own DjVu reader, written from the DjVu v3 specification: one page
or bundled documents, the ZP-coder, BZZ, JB2 masks with shared
dictionaries, IW44 backgrounds and foregrounds, palettes and hidden text.
PDF pages keep the layers (JPEG background, 1-bit masks) with the hidden
text as an invisible, searchable layer; pages also become PNG, JPG and
TIFF. Indirect documents are refused with their own message; sizes,
counts and unpacked bytes are bounded by DjvuLimits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…#346)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	docs/ENGINES.md
# Conflicts:
#	docs/ENGINES.md
# Conflicts:
#	README.md
#	docs/ENGINES.md
#	src/Filee.Core/Formats/FormatFields.cs
#	src/Filee.Core/Settings/AppSettings.cs
#	src/Filee.Core/Settings/UserDataStore.cs
@KnifeLemon KnifeLemon added the built-in engines Filee: built-in engines (Project 4), on the builtin-engines branch, not released yet label Nov 20, 2026
Some DjVu files, the DjVu v3 specification among them, give every word
zone the box of its whole line. Placed as is, the words of a line
overlapped in the PDF and text extraction interleaved their letters.
Lines whose word boxes don't follow each other left to right now get
their words spread over the line by length, like lines without words.

Checked on public DjVu files: pages render the same as the reference
pictures, and the specification's text now extracts in reading order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@KnifeLemon
KnifeLemon merged commit 5bca559 into builtin-engines Nov 20, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

built-in engines Filee: built-in engines (Project 4), on the builtin-engines branch, not released yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant