Skip to content

Chunked build: dependency-graph.json range-expansion is ~37 GB / ~7 min on the real models #32

@ebootheee

Description

@ebootheee

Finding (real-model regen, A1, 2026-05-29)

A clean parse of Outpost-A-1 (5.82M cells, 5.58M formulas) now completes after the
#31 emit/partition fixes, but the cell-level dependency-graph step dominates the
build
:

[chunked] Building dependency graph (streamed)... done — 5579816 formula cells (36.95 GB) in 411730ms
[chunked] ✅ Complete in 447.7s
  • 36.95 GB dependency-graph.json, 6.9 min (92% of the 7.7 min build).
  • Cause: write_dependency_graph (chunked_emitter.rs) uses the range-expanding
    extract_refs — every formula's range refs explode to ≤1000 cells each. On the
    monster sheets (PP&E 1.62M formulas, etc.) that's billions of edges.
  • RAM stays bounded (it's streamed to disk, ~2.3 GB), so it completes — but:

This is pre-existing (the range expansion came from the Round-2 is_cell_ref
change), not caused by #31 — it's only now visible because the build gets past
partition + emit for the first time.

Options

  • Scope the expansion to the cones of named inputs/outputs only (we don't need
    forward edges for every cell — only those feeding/affected by the contract cells).
  • Or stream-consume on the JS side (don't JSON.parse the whole file) when baking
    closures.
  • Or shard / compress, and make the full cell-level graph strictly opt-in.

Relates to #22 (output-cone scoping). Surfaced while validating #31.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions