Store prediction data in a pack file read per record; node + browser builds

[bkeepers]

Importing @neaps/tide-database eagerly parsed all 8,290 stations (via import.meta.glob({ eager: true })) into live JS objects — 118 MB heap / 663 MB RSS (the predictor itself is 4 MB). Two consequences:

• OOM on constrained devices — signalk-tides crashes a Victron Cerbo GX with a V8 Reached heap limit error (signalk-tides always causes signalk to restart after approx 30 minutes signalk-tides#103). It's not a leak: the prediction hot loop is flat over 600 iterations. The 118 MB baseline just consumes the constrained heap's headroom.
• Serverless cold-start CPU — the ~358 ms module parse dominates the tides API's per-request cost (~95 % of its Fluid Active CPU).

Solution: move the prediction data off the V8 heap

The OOM is a heapUsed vs --max-old-space-size limit. Anything bundled into JavaScript (object literals, JSON string literals, even a base64 literal → measured ~58 MB and OOMs under a 48 MB cap) lands on that heap. So the prediction data (harmonic_constituents, datums, epoch) ships as a file and is read by byte range instead.

• Build (scripts/generate-pack.mjs): writes stations.pack — the records concatenated as UTF-8 JSON — and a bundled byte-range index (id → [offset, length]). Metadata (identity + offsets/source/etc.) stays inlined as object literals. Offsets are UTF-8 bytes. Subordinate → reference links are validated at build time.
• Node build (station-data.ts): openSync the pack once; each lookup readSyncs only that record's range and parses it. Nothing resident — external stays ~2 MB, the OS page-caches touched pages, and the parsed record is transient. The read loops until the full range is filled (no uninitialized allocUnsafe bytes reach JSON.parse).
• Browser build (station-data.browser.ts): no filesystem, so the records are bundled as JSON strings and parsed on demand. The source is swapped per build behind the #station-data subpath import; exports conditions select dist/node vs dist/browser.

The public API is unchanged and still synchronous. Subordinate stations resolve to their reference's record. The MiniSearch text index is built lazily on the first search() and its serialized string is freed afterward. A build-time datums export replaces @neaps/api's all-stations datum scan.

Results

Node build, import + one nearest(), GC'd:

	heapUsed	external
eager (before)	118 MB	—
per-record pack (now)	35.8 MB	1.9 MB

The remaining ~36 MB is the sync API's own cost (8,290 Station objects + metadata + id maps), not the prediction data, which is never resident. Reads all 8,290 stations under --max-old-space-size=40; the old build needs ~69 MB just to start.

ESM only

kdbush and geokdbush are ESM-only packages that can't be require()d cleanly from a CJS build (double-wrapped default → KDBush.from is not a function), and all first-party consumers use ESM. So the package ships ESM only — the require condition is removed.

Testing

• All 31,862 tests pass; predictions and search unchanged.
• Deterministic builds: readAll() sorts by path; datums enum is sorted.
• A post-build smoke test (scripts/smoke.mjs, runs on every build) imports both built ESM entries, checks reference + subordinate resolution, and asserts the browser bundle contains no node:fs — a broken artifact fails the build.
• Tarball ships dist/node/generated/stations.pack (17.7 MB; package 9.7 MB).

Design notes and measurements: docs/lazy-loading.md.

Refs openwatersio/signalk-tides#103

[pkg-pr-new]

Open in StackBlitz

npm i https://pkg.pr.new/@neaps/tide-database@103

commit: 3a91ee9

[Copilot]

Pull request overview

This PR reduces baseline memory and import-time overhead by splitting station records into eagerly bundled “light” metadata and lazily parsed “heavy” fields (harmonics/datums/epoch), while also deferring construction of the MiniSearch text index until search() is called. It also adds a build-time datums export to avoid downstream runtime scans that would force parsing every station.

Changes:

• Introduces a build-time station bundling step (station-bundle.ts) to inline metadata as object literals and heavy fields as per-station JSON strings parsed on demand.
• Defers MiniSearch text index loading until first text search; geo index remains eager.
• Adds datums export + tests validating datum coverage and subordinate→reference lazy resolution behavior.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
test/station-bundle.test.ts	Adds coverage for new datums export and lazy heavy-field resolution.
src/types.ts	Introduces StationMeta / StationMetaKey types for the “light” station shape.
src/stations.ts	Switches station construction to meta + lazily parsed heavy fields; adds datums export.
src/station-bundle.ts	New build-time splitter that emits meta and heavy arrays and datum enum.
src/search/text.ts	Builds text index from station metadata only (no heavy-field touches).
src/search/index.ts	Defers text index loading until search() is called.
src/search/geo.ts	Builds geo index from station metadata only.
docs/lazy-loading.md	Documents the Phase 0 approach and longer-term lazy-loading proposal.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.