feat(excel): streaming reader for sheets too large for SheetJS

[aandre35]

Problem

A connector failed on a Lactel workbook with Sheet WHALY_test_2 not found in workbook even though the sheet exists. Root cause: that sheet runs to Excel's ~1,048,576-row ceiling (~2.3 GB of worksheet XML). SheetJS materializes each worksheet's XML as a single JavaScript string, and Node caps strings at ~0.54 GB, so SheetJS silently drops the oversized sheet from workbook.Sheets while keeping it in SheetNames. No amount of heap helps — it's a string-length limit.

Change

Adds an opt-in streaming Excel reader:

• .streaming() on ExcelExtractionConfigBuilder (single-sheet extraction only) + a streaming? flag on the config type.
• rowGeneratorFromExcelSheet() (excel/stream-reader.ts) parses the target worksheet row-by-row using yauzl (random-access zip — independent of archive entry order) + saxes (streaming SAX). It loads the small shared-strings table, then streams the worksheet, so memory stays bounded regardless of sheet size.
• Wires the streaming path into FileStream (no in-memory pre-count; totalRows stays undefined so progress logs omit the percentage).
• Preserves the in-memory reader's semantics: numberOfRowsToSkip, lowest-column "stop on empty" rule, type coercion, and derived (variableName) fields.

exceljs was evaluated first but rejected: its streaming reader crashes nondeterministically depending on zip entry order (workbook.xml position), which is unsafe for arbitrary client files.

Adds yauzl + saxes as dependencies; bumps the package to 0.4.0.

Verification

• New unit tests for the reader (cell types, skip, stop rule, derived fields, sheet fallback, missing-sheet error) and the builder flag. Full suite green.
• Ran the built reader against the real 2.3 GB sample with the production column config under a 1 GB heap cap: 1,043,058 rows in ~26 s, peak RSS 0.15 GB, correct values.

Follow-up

Consumers must bump to @whaly/connector-sdk@^0.4.0 once published. The Lactel file-uploads connector has been updated to call .streaming() on its circana_vmh config (separate repo).

🤖 Generated with Claude Code