P1+P2: runnable engine + artifact lock, schedules/drivers/fallback audit (#23–#26)#28
Merged
Conversation
… (P1 #23/#24) A clean `ete init` on the real PE models never finished: the chunked emitter built the cell-level dependency graph as a full in-memory map and then serialized the whole document into a second String, OOM-killing the parser on multi-million-cell models — and engine.js was emitted *after* that step, so the runnable engine never landed. The fixed 10-min spawnSync cap compounded it. Rust (chunked_emitter.rs): - Emit engine.js BEFORE the dependency graph (it depends only on the sheet DAG + partitions), so the runnable engine survives a kill of the later step. - Stream dependency-graph.json one entry at a time via BufWriter (write_dependency_graph) instead of building the full map + a second JSON String — the OOM fix. Schema unchanged (cell-dependency-edges-v1; edgeCount written last); output deterministic. init.mjs: - Configurable --timeout <seconds> (default 600->1800; 0 disables); the fixed 10-min cap killed legitimate large-model builds mid-emit. - Fail-loud: verify engine.js landed after a fresh parse before the manifest pipeline runs (--reuse-parse exempt, records the incomplete state instead). #24 (lib/build-manifest.mjs): - Emit chunked/build-manifest.json: locked artifact layout + a stable contentHash over the identity artifacts (engine.js, sheets/, _ground-truth.json, manifest.json). Derived contract maps are hashed for integrity but excluded from identity, so the hash is stable across rebuilds of the same workbook and changes on drift. Also the comprehensive completeness gate: a fresh build missing a required artifact hard-fails. --quiet now carries contentHash. Tests: new `npm run test:runnable` (+ CI step on ubuntu+windows). Full suite green: npm test 387, smoke 78/78, test:depgraph 11, test:engine 21, test:slimming 13, cargo test 11. Docs: CHANGELOG / PLAN / ROADMAP / README / HANDOFF. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… + opt-in gate Extends the contract maps for the Mippy calibration oracle. - Schedules/distributions (#25, Req A/B): pin per-year time-series outputs (distributionsToEquity, outstandingDebt, equityBase, freeCashFlow, and per-class classes[].distributions) into named-outputs.json as cellRange + perYear[], gated on manifest.timeline.columnMap. expandRange() brings them under the dependency closures (dependsOnNamedInputs / affectsOutputs). - Drivable named-inputs (#25, Req C): pin exitMultiple, exitYearSelector, hurdleRate (source: manifest-driver). Pinned in the normal init path; skipped under --reuse-parse without the workbook (follow-up). - Fallback audit (#26, Req D): scan generated sheet modules -> _fn-fallbacks.json; flag any named output/schedule whose dependency closure passes through an _fn() stub. Review-hardening (fixes a blocker in the gate as first written): - The gate threw mid-emit and `ete init`'s try/catch SWALLOWED it, silently dropping the whole contract (named-outputs/inputs/cell-types) while still reporting success — and on the real models (~11,813 fallbacks) it would fire routinely. It now emits all maps first, annotates affected outputs with `resolvesThroughFallback`, and records `stats.fallbackViolations`. - `ete init` surfaces violations: warns by default, hard-fails under `--assert-no-fallbacks` (CI / golden-master gate). Never swallowed. Tests: test-manifest-maps 47 -> 57 (audit reports without aborting, annotation, clean-build-stays-clean). Full matrix green: npm test 387, smoke 78/78, runnable 20, depgraph 11, engine 21, slimming 13, cargo 11. Docs: CHANGELOG / PLAN / ROADMAP / HANDOFF (gate described as report + opt-in). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #23, #24, #25, #26. Two commits on this branch.
P1 (#23 + #24) — reliably emit a runnable
engine.jsA clean
ete initon the real models never finished: the chunked emitter built the cell-level dependency graph as a full in-memory map + a second serializedString(OOM on multi-million-cell models), andengine.jswas emitted after it, so the runnable engine never landed.engine.jsbefore the dependency graph (it depends only on the sheet DAG + partitions); streamdependency-graph.jsonto disk one entry at a time (the OOM fix). Output stays deterministic; schema unchanged.ete init --timeout <seconds>(default 1800;0disables); verifyengine.jsafter a fresh parse (fail loud, never partial;--reuse-parseexempt).chunked/build-manifest.json(Artifact must be consumable downstream without per-version reconciliation (lock layout + emit content hash + always emit engine.js) #24): locked artifact layout + a stablecontentHashover the identity artifacts (engine.js, sheets/, _ground-truth.json, manifest.json) — stable across rebuilds, changes on drift. Newlib/build-manifest.mjs,npm run test:runnable+ CI step.P2 (#25 + #26) — contract schedules, drivable inputs, fallback audit
distributionsToEquity,outstandingDebt,equityBase,freeCashFlow, per-classclasses[].distributions) pinned intonamed-outputs.jsonascellRange+perYear[], brought under the dependency closures viaexpandRange().exitMultiple,exitYearSelector,hurdleRatepinned intonamed-inputs.json._fn-fallbacks.json; flag any named output/schedule whose closure passes through an_fn()stub. Reports by default (annotates outputs withresolvesThroughFallback+stats.fallbackViolations;ete initwarns); hard-fails under--assert-no-fallbacks(CI / golden-master gate) so it doesn't block the real models (~11,813 fallbacks today).Review-hardening
The #26 gate as first written
throwew mid-emit andete init's try/catch swallowed it — silently dropping the entire contract (named-outputs/inputs/cell-types) while still reporting success, a failure that would fire routinely on the real models. It now emits all maps first, then surfaces the result so it can never be swallowed (reproduced + verified end-to-end).Tests
Full matrix green:
npm test387 (test-manifest-maps 47→57), smoke 78/78,test:runnable20,test:depgraph11,test:engine21,test:slimming13,cargo test11/11 (0 warnings). CI runs ubuntu + windows.Known follow-ups
--reuse-parsewithout the workbook are skipped (work in the normal init path).baseCaseValuesums across years (right for flows, not balances;perYearis authoritative).🤖 Generated with Claude Code