Skip to content

Built-in engines W29: e-book and markup writers, UYAP UDF writer, interactive whiteboards - #1723

Open
KnifeLemon wants to merge 10 commits into
builtin-enginesfrom
p4/w29
Open

KnifeLemon wants to merge 10 commits into
builtin-enginesfrom
p4/w29

Conversation

@KnifeLemon

Copy link
Copy Markdown
Owner

Wave 29 of the built-in engines: e-book and markup writers without calibre and Pandoc, a UYAP UDF writer, and interactive whiteboards.

E-book writers without calibre

  • New writers in Ebooks/Legacy/ from the shared Book model: PDB (PalmDOC, 4 KB records with the existing compressor, bookmark records for headings 1–3), TCR (dictionary built per book), PML (formatting, links, footnotes, pictures in a <name>_img folder), RB (zlib-chunked HTML, info page, PNG pictures), SNB (bzip2 text blocks with Filee's own encoder, zlib file table).
  • Windows-1252 when every character fits, else UTF-8; PML is ASCII with escapes.
  • Every book reader and HTML/Markdown/TXT/DOCX → these five at cost 19, like AZW3/MOBI; calibre still wins when ranked first.
  • RocketBookReader reads the info page as UTF-8 when valid (was Latin-1).
  • Limits through DocumentLimits (text, records, files), cancellation, WriterTempFile.
  • eReader PDB isn't written (Filee's reader refuses it, so nothing to round-trip against).

Markup writers without Pandoc

  • LaTeX, AsciiDoc, Typst and man (roff) writers on the shared MarkupTextWriter walk: headings, emphasis, nested lists, quotes, code, tables with merged cells, links, footnotes, pictures (not in man), rules; full escaping per language.
  • LaTeX compiles with pdfLaTeX, XeLaTeX or LuaLaTeX; xeCJK/luatexja when CJK text is present.
  • MarkupConverter.WriteOnly: every built-in reader's format → tex/adoc/typ/man at cost 8 (Pandoc 11). Reading these four still uses Pandoc.
  • Equations: new TexMathWriter (LaTeX, AsciiDoc latexmath, Markdown) and TypstMathWriter. Markdown now writes Word/한글/MathML equations as $…$ / $$…$$ and reads them back as the same equation.
  • Not compiled or rendered with TeX, Typst, Asciidoctor or groff (not installed here).

UYAP UDF writer (#1546, #1633)

  • UyapWriter / udf-writer: every built-in reader's format → UDF in one step at cost 8. Paragraphs (alignment, indents, spacing, tabs, lists), runs (font, size, styles, colour), tables (widths, colspan, fill, borders, header rows; vertical merges drawn with borders), pictures, one header and footer, page breaks, A4 portrait/landscape.
  • CDATA split at ]]>, invalid XML characters and \r dropped so offsets stay right; content checked against the reader's limits; very repetitive content stored to pass the reader's inflate-ratio check.
  • Checked against the [Format]: .udf #1338 sample (not added): sample → DOCX → UDF keeps the text exactly, all 65 paragraph property sets and 84 of 85 runs.
  • UyapReader now reads lists, cell alignment/fill/borders and header rows.
  • UDF → XLSX/ODS/CSV through the document tables engine. UDF → PNG/JPEG already worked through PDF (now tested).
  • udf → folder is DirectOnly, so a document isn't routed through UDF to be unpacked.

Interactive whiteboards (#1665)

  • IwbReader (IEC 62874 / Becta IWB, OpenBoard spellings) and SmartNotebookReader (.notebook) translate the SVG pages into the slide model with SvgSlideReader: shapes, paths, gradients, text and text areas, XHTML text, pictures, use.
  • PDF (selectable text), SVG and PNG through the slide renderer; ODP through OdpWriter (editable); PPTX through the deck's PDF and PdfSlidesConverter.
  • Audio, video and undecodable pictures in the left-out notice.
  • Limits on the package, XML, depth, use count, visits, path points, objects and pages (error.document_too_complex); fuzz target IwbFuzzTests.
  • Settings schema v115: the presentations donut learns .iwb and .notebook. General grade count 109.
  • Not tried on a real IWB or Notebook file ([Format]: .iwb #1665 has no attachment).

Merge notes

  • docs/ENGINES.md: the E-books row from the e-book branch, the Drawings row from the whiteboard branch.
  • Settings schema 115 (113, 114, 116 unused).

Tests

Run locally on the merged branch:

  • Release build
  • Filtered engine tests (legacy e-books, Kindle, KEPUB, markup, TeX math, Markdown, Pandoc, UYAP, document tables, registry, archive, routes, IWB, drawings, ODF draw, DjVu, RTF writer, fuzz): 677 passed, 27 skipped, 0 failed
  • Core: 1796 passed, 11 skipped
  • App localization, settings and IWB tests: 280 passed, 9 skipped
  • dotnet format --verify-no-changes --severity warn clean on Engines, Core and their tests

🤖 Generated with Claude Code

KnifeLemon and others added 9 commits November 20, 2026 21:02
The built-in e-book engine now writes the older formats it already read,
from the document model the other writers use:

- PDB as PalmDOC: 4 KB PalmDOC-compressed text records, the PalmDOC
  header, the title as database name, a bookmark record per heading
- TCR: a 256-entry dictionary (every byte used plus frequent pairs
  merged on a sample), the text coded by longest match
- PML: ASCII with \aNNN / \UXXXX, chapters and sub-headings, styles,
  alignment, list markers, links and anchors, footnotes, PNG pictures in
  the <name>_img folder reserved through the output allocator
- RB: the XHTML page in zlib chunks, the info page, PNG pictures
- SNB: book, contents and chapter XML, bzip2 text blocks (own encoder),
  file table and block tail

Text is Windows-1252 when it fits, else UTF-8; the RB reader now reads an
info page in UTF-8 too. Each writer stays within its reader's limits,
checks for cancellation and writes through a temporary file. The edges
cost 19 like AZW3 / MOBI, so calibre ranked first still takes EPUB to
these formats.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- LatexWriter, AsciiDocWriter, TypstWriter and ManWriter on the shared
  MarkupTextWriter walk: headings, emphasis, nested and numbered lists,
  quotes, code, tables with merged cells, links, footnotes, pictures,
  rules, and every special character of each language escaped.
- MarkupConverter.WriteOnly offers every built-in reader's format into
  tex, adoc, typ and man at cost 8, below Pandoc's 11; reading them stays
  Pandoc's. Output goes through a temporary file; the walk checks the
  conversion's cancellation between blocks.
- TexMathWriter writes the equation tree (Word math, 한글 equations,
  MathML) as TeX for LaTeX, AsciiDoc and Markdown, so Markdown output no
  longer writes linear text for them; TypstMathWriter writes Typst math.
  Trees deeper than 256 levels keep their text.
- Tests build their models in code; the hostile writer tests cover the
  four new writers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Anything Filee reads (Word, ODT, RTF, HWP/HWPX, Markdown, HTML, PDF,
e-books, ...) now converts to UDF, the UYAP Document Editor's file, in
one step through the new udf-writer engine. The writer works from the
document model like the RTF and ODT writers and writes content.xml the
way the editor does: one CDATA text with the header, body and footer,
runs and pictures by offset, the page format, paragraphs with
alignment, indents, spacing, line spacing, tab stops and lists, runs
with font, size, bold/italic/underline/strike and colours, tables with
column widths, colspan, fill, alignment and cell borders, pictures as
PNG or JPEG, page breaks, one header and one footer.

- Text XML can't hold is left out without moving the offsets; "]]>"
  splits the CDATA block.
- content.xml stays within what the reader takes (part size, elements,
  paragraphs, objects, inflate ratio); pictures past the limits are left
  out; a refusal, failure or cancellation leaves no file.
- UDF becomes writable. The reader now reads lists (Numbered/Bulleted,
  ListId, ListLevel), header rows, row heights and the cells' align,
  fillColor, borderNone and borderSpec.
- UDF tables go to XLSX, ODS and CSV through Document tables; pictures
  of the pages come from the Filee renderer's PDF.
- UDF -> folder (Extract) is taken only from a .udf itself, not from a
  document written as UDF on the way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… ODP and PPTX (#1665)

Filee's own reader of interactive whiteboard files (IEC 62874 Common File
Format): the pages of content.xml's SVG (pageSet, also written pageset)
become slides of the drawing's size, with what the svg holds outside the
pageSet on every page and iwb:element invisible elements hidden. SMART
Notebook files (a page per SVG file, in imsmanifest.xml's order) are read
the same way.

The SVG is read into the slide model by SvgSlideReader: shapes and paths
with their transforms applied, fills, strokes, gradients, text and
textArea, XHTML text in foreignObject, pictures from the package or data:
URIs, SVG pictures, use and nested svg. The drawing engine draws them to
PDF, SVG and PNG, writes ODP with the ODP writer and PPTX through the PDF
slides engine. Videos, sounds and pictures it can't decode are named in the
left-out notice. Package, XML, nesting, uses, visits and path points are
bounded; hrefs never leave the package.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tebook, #1665)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	docs/ENGINES.md
@KnifeLemon KnifeLemon added the built-in engines Filee: built-in engines (Project 4), on the builtin-engines branch, not released yet label Nov 20, 2026
Where Pandoc is installed (CI), the document engines read LaTeX, AsciiDoc,
Typst and man through Pandoc's AST, so a route to Markdown exists without
the Pandoc converter. The test now checks what it means to: the built-in
markup engine never reads these formats.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

built-in engines Filee: built-in engines (Project 4), on the builtin-engines branch, not released yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant