Skip to content

Commit 188c268

Browse files
authored
Unify worksheet drawing objects into one z-ordered list (#19)
* Unify worksheet drawing objects into one z-ordered list Replace the per-kind worksheet storage (charts, chartEx, images, form controls, comments, raw drawing XML) with a single drawings list whose order is z-order, matching the drawing part document order in XLSX/XLSB, the OfficeArt container order in XLS, and the one-Shapes- collection model Excel itself exposes. Shared non-visual properties (name, hidden, locked, printable, alt text, title) hoist into DrawingMeta on the wrapper; payload types lose their anchor/name/description fields. Comments join the list (popup visibility is the wrapper hidden flag) while keeping the keyed (row, col) accessors as views. Shape groups get a tree representation with child-space transforms, path-based node access, and flattened traversal; form-control sync and radio grouping operate on placed controls with resolved rectangles so grouped controls keep working. Filtered typed views (charts(), images(), form_controls(), yielding the wrapper alongside the payload) replace the per-kind slices, and kind-scoped positional mutators are removed in favor of the single drawing index space. Readers append in file order per kind and writers consume the views, so emitted bytes are unchanged. * Preserve XLSX drawing z-order, groups, and shared shape flags Read the drawing part in document order with single ownership per anchor, fixing the image duplication where every pic was stored both parsed and as a raw anchor. Form controls join the z-order through their a14 placeholder twins (MS-ODRAWXML Legacy Object Wrapper), which the writer now emits with matching shapeId/spid/name across the worksheet controls block, the VML part, and the drawing part; Excel opens the result without repair and keeps the control position on re-save. Shape groups round-trip as a modeled tree with child-space transforms instead of degrading to raw XML. Comments splice into the list by their legacy VML sequence relative to controls, carry their popup anchor through the wrapper instead of being re-synthesized from the cell, and cross-layer order against native objects normalizes on round-trip. Raw anchors keep a parsed placement, capture their relationships and referenced parts so rewrites do not dangle, and hold their document position. Image and chart locked/print flags round-trip through xdr:clientData. * Preserve XLS drawing z-order, shape groups, and FOPT names Walk the OfficeArt shape tree in true pre-order so SP containers pair with their OBJ records even when SpgrContainers are present; the old deferred-children walk mispaired every shape after a group. Build the drawings list in container order across pictures, comments, controls, and groups, and emit it back in the same interleaved order, so BIFF8 round-trips full cross-kind z-order. Shape groups round-trip as a modeled tree: OfficeArtFSPGR carries the child coordinate space and OfficeArtChildAnchor the child rectangles, with group OBJ records (FtCmo ot 0x00 + FtGmo) per MS-XLS 2.4.181. Control and picture shape names and alt text round-trip through FOPT wzName/wzDescription, and picture locked/print flags through the ftCmo grbit. * Write XLSB drawings from the model and fix comment record ids The XLSB writer previously replayed an opaque multi-part bundle captured at read time and never emitted model drawings: images and charts added through the API vanished, chart edits were silently discarded, and a dangling BrtDrawing pointer made Excel refuse the file. Extract the drawing-part codec and chart XML writers from the XLSX crate into duke-sheets-chart (parse/write features) and drive both formats from the drawings list: document-order emission, groups, raw anchors with captured rels, and control placeholder twins in the format Excel uses per package flavor (sp/a14:compatExt for XLSX, graphicFrame/com14:compatSp for XLSB, pinned against Excel output). Control names now ride the XLSB twin cNvPr and BrtDrawing (0x0226) joins BrtLegacyDrawing in the sheet stream. The comments part had shipped with off-spec record ids (0x0278-based instead of the 0x0274-based sequence in MS-XLSB 2.4.33) and a 12-byte BrtBeginComment; every in-process round-trip passed because reader and writer shared the bug, while Excel refused every comment-bearing XLSB we ever wrote. Emit the spec sequence (36-byte BrtBeginComment with UncheckedRfX and GUID, BrtCommentText rich string) and keep reading the legacy ids for files written by older versions. Excel COM parity now covers XLSB comments, images, charts, and z-order, plus XLSX cell comments. * Round-trip drawing visibility across XLSX, XLSB, and XLS Carry DrawingMeta.hidden through native DrawingML cNvPr properties for pictures, charts, and groups (including group children), and through legacy VML visibility styles for form controls. Control twin cNvPr hidden remains structural and is deliberately independent of the user-facing control visibility. For BIFF8, parse and emit the Group Shape Boolean Properties FOPT bag (opid 0x03BF): fHidden is bit 1 and only counts with its matching high-word use bit per MS-ODRAW 2.3.4.44. Excel round-trip parity now pins hidden pictures and form controls in all three formats. * Add rich form-control captions, alignment, and macro metadata Replace plain caption strings with ControlText, the single source of truth for rich runs plus optional horizontal and vertical alignment. Round-trip run fonts (family, size, color, bold, italic, underline and other supported properties) through DrawingML txBody for XLSX, VML for legacy/XLSB controls, and BIFF8 TXO formatting runs backed by real FONT records. Plain captions remain one unformatted run and preserve the existing per-kind visual defaults. Carry control macro assignments through XLSX control properties, XLSB VML FmlaMacro, and XLS FtMacro formulas referencing fProc Lbl records. Complete control name/alternative-text/title carriers through native cNvPr/controlPr/VML/FOPT where each format supports them (XLS has no verified title carrier). Focused Excel parity verifies rich text, alignment, metadata, and macro names survive re-save in all three formats. * Model and round-trip basic worksheet shapes Add first-class Shape objects to the unified drawing tree with preset geometry, solid/no fill, line color/width/dash, rotation/flips, and DrawingText rich runs plus alignment. Shapes participate in z-order, groups, typed worksheet views, and shared DrawingML emission for XLSX and XLSB. Preserve unsupported OOXML shape-property/text fragments on untouched round-trips while parsed snapshots keep modeled edits authoritative instead of replaying stale XML. Add BIFF8 rectangle, round-rectangle, ellipse, triangle, and line support using their pinned OfficeArt FSP types, ordinary OfficeArt OBJ hosts, FOPT fill/line/transform metadata, and TXO text runs. Excel may normalize generic OfficeArt OBJ type 0x001E to its specific line/ rectangle/oval/text/polygon types on re-save; the reader accepts that full shape family. Unsupported XLS preset names fail explicitly rather than silently becoming rectangles. Focused Excel parity verifies full shape survival in all three formats. * Preserve unknown controls and use worksheet metrics for anchors Surface unsupported legacy Forms controls as typed placeholders carrying their object type, rich caption, raw control/VML properties, and BIFF OBJ body where available. EditBox/Dialog content round-trips through XLSX and XLSB; XLS safely replays only captured OBJ bodies and rejects new unknown records rather than guessing. ActiveX/OLE VML shapes (including ObjectType=Pict) are explicitly excluded from this path and linked-cell synchronization ignores unknown semantics. Add DrawingMetrics with Excel-compatible column-width and row-height conversion, implement fast prefix positions on Worksheet, and resolve OneCell/Absolute control/VML/BIFF anchors against actual sheet sizes. Radio-group placement now uses custom row and column dimensions too. Focused Excel parity pins marker preservation in XLSX, XLSB, and XLS. * Expose the unified drawing tree in Node, Python, and WASM Replace binding-level per-kind positional mutation with one recursive, z-ordered drawing API. Every drawing carries a stable drawingPath, shared metadata, top-level anchor or child transform, and a typed payload for images, charts, controls, comments, shapes, groups, or raw placeholders. Add top-level insert/move plus nested set/remove, lazy image byte accessors, and flattened path-aware form-control/image views without eagerly copying media bytes. Expose control-to-cell and cell-to-control synchronization in every binding. Add a semantic check-state interaction: checking a radio unchecks its spatial group siblings and synchronizes linked cells immediately; raw setDrawing remains a structural replacement. Remove the ambiguous add/set/removeFormControl index API. Update the manual TypeScript unions, Python classes/exports, WASM custom declarations, and focused binding tests. * Document unified drawings and link format-specific parity coverage Update the canonical feature matrix for drawing z-order, groups, visibility, basic shapes/text boxes, nonvisual metadata, rich form- control text/alignment, macro assignment, unknown Forms placeholders, and custom-metric anchors. Link every new W✔ claim to a per-format round-trip and Excel parity test; keep groups, unknown controls, comment anchors, and XLS title metadata partial where dedicated parity or a format carrier is missing. Correct stale claims that XLSB had no image/drawing support and that controls lacked native twins, names, rich captions, or macros. Add and pin XLSB one-cell/absolute image anchor survival through real Excel, and update exhaustive drawing-kind test helpers for first-class shapes. * Make feature support claims evidence-driven Audit every format cell in FEATURES.md against the tests it links. Relink 299 exact per-format/parity tests, fix 64 dead, shorthand, mispathed, or wildcard references, and correct the stale top-level format table. Downgrade claims that lack format-specific reader/writer coverage or Excel parity instead of allowing symmetric codec bugs to appear fully supported: 166 read cells become unverified and 457 write cells become partial/unverified. XLSB has the largest correction. Add a standard-library checker that inventories Rust tests, resolves exact links, classifies read/write format coverage, requires canonical Excel parity for every W✔, and requires limitation notes for partial cells. It now passes over 392 format rows and 1,007 supported directions. Add the missing in-process XLS custom-metric anchor case so that parity-backed claim remains fully supported despite BIFF offset quantization. * Fix XLSB visual conditional-format records and deepen parity coverage Excel refused advanced conditional-format XLSB files even though our reader/writer round-trip passed: both codecs shared wrong rule template ids, CFVO enums/offsets, color encoding, and malformed data-bar/icon- set records; rules also shared merged ranges and duplicate priorities. Emit and parse the MS-XLSB layouts pinned against Excel output, isolate ranges per rule, and assign unique priorities. A byte-level regression pins the exact BIFF12 record sequence and payload fields. Split a high-risk parity probe into deep Excel survival tests for advanced conditional formatting, range-list validation messages and warning alerts, split panes, and sheet permission/password flags. Also strengthen table parity to cover first/last-column and both stripe flags. Update FEATURES evidence: XLSB gains nine fully parity-backed writer claims while unsupported variants remain partial. * Tighten drawing-model semantics from review findings Scope the interactive check-state sync to the touched control and its radio-group siblings; Excel never rewrites unrelated stale links on a click, and the previous workbook-wide sweep could. Shape equality now tracks emission semantics (modeled fields, preserved bytes, and the staleness outcome of each preserved block), preserved raw XML is validated as balanced well-formed fragments (text bodies must be a single txBody) instead of being spliced verbatim, and unbalanced fragments fail the write rather than truncating silently. Comments are rejected as group children, matching every format. Off-grid negative anchor coordinates now clamp to the grid origin instead of wrapping to the far edge of the first cell, and degenerate zero-extent grids resolve to the first row/column of a tie. Empty comment authors keep a distinct author slot in XLSX and XLSB so anonymous comments are not attributed to author 0, with the XLSB writer sharing the core derivation. Generated cNvPr ids seed past ids already used inside raw passthrough anchors. Remove the unused default-metric wrapper functions, declare the chart-crate features core needs to build standalone, and repoint tests that pinned textboxes as unmodeled content at connector shapes now that textboxes parse as first-class shapes. * Preserve unknown-control passthrough and comment uniqueness in JS bindings Node and WASM setDrawing rebuilt Unknown form controls with empty raw properties, client data, and OBJ bytes, silently destroying the passthrough data that placeholder kind exists to preserve; the snapshot DTOs now carry the raws as documented opaque fields that round-trip through setDrawing. Both bindings also skipped the one-comment-per-cell invariant on drawing replacement (and on group inputs nesting comments); they now enforce it like Python. Fix the WASM serializer to emit plain objects instead of ES2015 Maps for flattened drawing DTOs, matching the declared TypeScript types. * Harden the feature-matrix checker against false passes Make malformed rows fatal (wrong cell counts, more than one Notes overflow cell, duplicate directions in a status cell), require body evidence of reader/writer/round-trip calls instead of granting directions from test names, and credit parity read-back only to the Excel round-trip helpers rather than any binding named result. * Remap conflicting relationship ids across raw drawing fragments The drawing-rel planner merged raw fragments' preserved relationships first-wins by id: when two fragments reused one rId for different targets, the second entry was dropped from the .rels part and the second fragment's references silently resolved to the first target. Conflicting reuses now get a freshly allocated id, rewritten inside that fragment's bytes (quote-delimited so rId1 never matches inside rId10); same-target reuses keep sharing the original id. Raw group children, which the planner previously skipped entirely, now participate in reservation and merging, and the preserved-part collectors dedup by target instead of id so a remapped rel's part bytes still land at their path. * Treat hidden rows and columns as zero-extent in drawing metrics Excel renders hidden rows and columns at zero extent, so anchors flattened through worksheet metrics landed past their on-screen position whenever hidden rows or columns sat inside the span. Hidden rows/columns now report zero width/height and contribute no position advance. All fraction and offset conversions already guard zero extents (XLS 1024ths/256ths return 0, column_at_emu/row_at_emu resolve zero-extent runs to their first index). * Apply group rotation and flips when mapping child rects placed_form_controls resolved group children through a purely axis-aligned chOff/chExt mapping, so controls inside rotated or flipped groups got worksheet rects on the wrong side of the group frame, misassigning radio groups and check-state hits. Child corners now mirror (flips) and turn (rot, 60,000ths of a degree clockwise) about the outer frame center in DrawingML transform order, and the mapped rect is the bounding box of the transformed frame. A child shape keeps its unrotated frame, matching how top-level rotated shapes anchor; nested group rotation composes through recursion. * Share legacy VML part assembly between XLSX and XLSB writers The vmlDrawing part builder (shapelayout/idmap preamble, note and control shapetypes, drawing-list walk with comment/control shape-id blocks, the Note shape emitter) and the SheetControl sequence helpers were byte-for-byte duplicated across the two writers, differing only in zip plumbing. Both now delegate to build_legacy_vml and the sheet-control helpers in duke-sheets-vml, which already owned all shape-level emission; each format keeps its own part path, zip options, validation policy, and rels decision. * Add feature relevance markers to the FEATURES.md checker The checker verifies that a linked test exercises the right format but not that it exercises the row's feature, so a wrong link that happens to touch the format passes silently. Tests can now declare the rows they evidence with a "// features: Row Name; Other Row" comment above their attribute block; when a linked test carries markers, every row linking it must be named, and markers naming no existing row fail the check. Unmarked tests stay exempt so adoption is incremental. Annotate the drawing, shape, hidden-flag, control interaction, and unknown-control test suites as the starter set (21 tests, 31 links relevance-checked). * Fix caption conversion in XLS form-control round-trip test * Make empty drawing text convert to no runs DrawingText::plain wrapped empty strings in a single empty run while readers produce no runs for absent captions, so empty captions never compared equal across a round trip. Update the XLS mixed-state test to the core validation message that now fires before the writer's own check. * Update XLS tests for palette mapping and exact anchor quantisation The font-color test pinned the old always-Auto fallback; the writer now maps palette-exact RGB to its indexed slot, so pin exact survival for palette colors and Auto fallback for off-palette values, and update the FEATURES link and note. The picture-offset test assumed truncated 595/744 EMU-per-unit constants; anchor conversion now uses exact rational 1024ths/256ths, so use offsets exact under that quantisation. * Accept Uint8Array byte payloads in WASM drawing inputs Drawing inputs deserialize through serde's internal buffering (tagged enums and flatten), which captures a Uint8Array as bytes; plain Vec<u8> fields only accept sequences and rejected them with "invalid type: byte array". Byte fields now deserialize through a visitor accepting bytes or sequences. * Keep anchor resolution inside the first visible cell after a hidden run The zero-extent tie resolution walked back to the run head for any position sharing the run's start, so a point strictly inside the first visible row or column after hidden ones collapsed to the hidden run head at offset zero. Flattened anchors lost up to a full cell of extent and OneCell anchors could come out reversed (to < from), which flows into XLS ClientAnchor, legacy VML, and control-twin emission. Walk back only when the position sits exactly on the shared boundary, where canonicalizing to the run head keeps degenerate metrics stable. * Count group-nested controls when gating XLSB legacy VML records The sheet part gated BrtLegacyDrawing (and the BrtDrawing rid offset) on the top-level form-control count while the VML part and the sheet rels follow placed controls, which include group children. A sheet whose only control lives inside a group emitted the VML part and its relationship but no BrtLegacyDrawing, and BrtDrawing pointed at the vmlDrawing relationship: a dangling drawing pointer Excel refuses. The reader resolves parts by relationship type, so only a bytes-level check of the rid pairing catches it; the new test asserts the pairing and that the VML carries the nested control shape. * Ignore twin text on caption-less controls instead of panicking The XLSX reader unwrapped caption_mut() whenever a matched control twin carried a txBody, but scrollbars, spinners, list boxes, and dropdowns have no caption; a third-party file with stray twin text crashed the reader. Permissive readers must not panic on untrusted input; the stray text is now ignored at both assembly sites (top-level twins and group children). * Fix inverted fLine polarity on XLS basic shapes basic_shape_fopt emitted fLine=0 for outlined shapes and fLine=1 for no-outline ones, and the reader inverted the same bit back, so every round-trip stayed green while Excel rendered outlined shapes without their outline (line-geometry shapes became invisible) and Excel-authored outlines read back as no_fill, destroying them on conversion to other formats. MS-ODRAW 2.3.8.44: fLine (0x00000008) displays the line; fUsefLine is 0x00080000. The control and picture FOPT paths already used the correct polarity. * Regenerate stale Node generated.d.ts The committed napi output predated the unified drawings branch: it still declared the removed addFormControl/setFormControl/ removeFormControl methods, the formControls getter, JsFormControl interfaces, and anchor fields on JsChart/JsChartEx, and index.d.ts re-exports it wholesale, so the published type surface advertised APIs that fail at runtime. * Reconcile controls sharing a linked cell after interaction The scoped interaction sync updated only the toggled control and its radio-group siblings, so a checkbox or radio group elsewhere linked to the same cell kept its stale state, and the save-time full sync projected that stale state back over the user's change (last control in worksheet order wins). After the scoped projection, checkboxes and radio groups whose links resolve to a touched cell are now driven from the new cell value, as in Excel, where every control sharing a linked cell follows it. Also adds the ChartSheet.raw_drawing_rels field capturing relationships referenced by preserved chartsheet anchors (consumed by the XLSX reader/writer). * Harden the core drawing model against review findings set_comment_at now validates row/col bounds like every other comment mutation path and returns Result (a malformed key previously emitted an out-of-grid ref in the comments part). validate_kind rejects Raw group children: raw bytes are whole anchor elements and cannot nest inside grpSp. map_child_rect uses saturating math so hostile metrics cannot overflow-panic on save. default_comment_anchor flips the popup to the left of cells near the right grid edge instead of emitting a zero-width anchor, matching Excel. Document the empty-caption convention (DrawingText::plain("") is zero runs) on DrawingText and RichTextRun. * Fix XLS drawing wire-format findings from review Groups and pictures wrote FOPT rotation in raw 60,000ths of a degree where MS-ODRAW specifies 16.16 fixed-point degrees (basic shapes already converted), so a 90-degree rotation rendered as 82.4 in Excel and Excel-authored rotations read back wrong, feeding bad angles into group-child mapping. FtMacro now sits after cbls/rbo/sbs per MS-XLS 2.4.181 instead of directly after ftCmo. Macro Lbl indexing counts only names that will be emitted, and unemittable macro names fail the write with a clean error instead of silently desyncing every later PtgName. Unknown-control replay strips the embedded FtMacro (whose rgce pointed at the source workbook Lbl table) and emits a fresh one. Fills and borders palette-match exact RGB like the font path, with Indexed unified on raw icv. Caption runs with ifnt 0 read back with no font instead of an explicit default; shape text alignment defaults strip to None like control captions. Comment popups emit their model anchor instead of always the synthesized default. ClientAnchor fractions clamp at zero per MS-XLS 2.5.193. The oversized-caption test now actually spans multiple CONTINUE records, and a new test pins picture anchors over hidden rows and columns. * Fix OOXML drawing pipeline findings from review XLSB non-visual conditional-format rules emitted invalid CFType values (7..17, where MS-XLSB 2.5.18 defines only 1..6) with wrong templates; every predicate now rides CF_TYPE_EXPRIS with its CFTemp template (MS-XLSB 2.5.16, 2.4.23), the reader decodes by template, and legacy off-spec ids remain readable. Charts keep their parsed oneCell/absolute anchors and editAs instead of collapsing to a degenerate twoCell. Raw-fragment relationship capture matches any attribute resolving to a rel id (SmartArt dgm:relIds, o:relid) rather than a hardcoded attribute list, and the planner reserves ids referenced only inside raw bytes. wsDr-level mc:Choice captures inject the wrapper's xmlns declarations into preserved fragments so splicing them back stays well-formed. Chartsheet raw anchors carry their captured relationships through the new core field and the chart rid allocates around preserved ids instead of hardcoding rId1. Comment Note shapes emit x:Visible when shown, which Excel keys the shown state on. The XLSB comments dialect latch skips BrtACBegin wrappers; the comment GUID version nibble sits in the correct byte; the chartEx content type matches Excel's lowercase form; rel-id rewriting is attribute-anchored so text nodes are never rewritten; and a malformed drawing part is skipped permissively instead of failing the whole workbook. * Close feature-checker false-pass vectors FileFormat::X mentions no longer grant writer evidence on their own; line comments are stripped before body evidence matching (so commented-out round_trip calls and format mentions grant nothing); the static file whitelist supplies only the format tag with directions requiring body evidence; the scanner discovers single-line attribute+fn tests and multi-line cfg_attr attributes; duplicated feature names across sections get a non-fatal note. FEATURES rows gain links and notes for the new XLS comment-anchor, palette-color, macro-indexing, unknown-control replay, and group-rotation tests. * Keep placeholder FBSE slots so blip ids resolve positionally pib blip ids are 1-based positions over every FBSE in the BSTORE, but the reader only stored entries with embedded blips, so a header-only placeholder shifted every later picture onto the wrong blip or none. Excel emits exactly such placeholders when it rasterizes picture transforms while saving XLS: with the writer now emitting spec 16.16 rotation values, a quarter-turn picture comes back from Excel as a placeholder FBSE plus a re-encoded bitmap, and the whole picture vanished on read. The parity test previously passed only because the old writer wire value made 90 degrees look like 82.44 to Excel, an odd angle it passes through. Pin the real oracle behavior: non-quarter-turn rotations (and their flips) survive as live transforms, quarter-turn rotations and pure flips rasterize into the pixels with identity transforms on read-back. * Align binding drawing APIs on native anchors and honest types Node and Python flattened every anchor to two-cell at default metrics on read and only accepted two-cell input, so a get -> set of a drawing read from a file with a oneCell or absolute anchor silently rewrote the variant and baked default-metric coordinates, visibly moving drawings on sheets with custom column widths or row heights. Both now expose the tagged anchor union WASM already used, both directions. Python DrawingMeta.hidden becomes Optional and defaults per kind like JS, so passing meta no longer forces comments visible; unknown controls gain raw passthrough getters. Chart input TS types narrow to the fields the deserializers accept instead of advertising silently dropped ones; option-button output state admits the mixed value permissive readers can surface; raw-drawing metadata carries full relationship descriptors in Node like the other bindings; byte inputs document Uint8Array acceptance and setDrawing rejects comment and raw group children instead of inserting invalid trees. Stale-path hazards are documented on the path-taking APIs.
1 parent 79bf8ba commit 188c268

113 files changed

Lines changed: 35872 additions & 10285 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

FEATURES.md

Lines changed: 270 additions & 262 deletions
Large diffs are not rendered by default.

bindings/nodejs/Cargo.lock

Lines changed: 15 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

bindings/nodejs/Cargo.toml

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,8 +15,10 @@ duke-sheets-core = { path = "../../crates/duke-sheets-core" }
1515
duke-sheets-xlsx = { path = "../../crates/duke-sheets-xlsx" }
1616
duke-sheets-csv = { path = "../../crates/duke-sheets-csv" }
1717
duke-sheets-chart = { path = "../../crates/duke-sheets-chart" }
18-
napi = { version = "3", default-features = false, features = ["napi9", "tokio_rt"] }
18+
napi = { version = "3", default-features = false, features = ["napi9", "tokio_rt", "serde-json"] }
1919
napi-derive = { version = "3", features = ["type-def"] }
20+
serde = { version = "1", features = ["derive"] }
21+
serde_bytes = "0.11"
2022

2123
[build-dependencies]
2224
napi-build = "2"

0 commit comments

Comments
 (0)