Make the futures provider actually runnable: norgate extra, volume=, and the port verification

[mspinola]

Follow-up to #7, which merged at 340b2ef — the provider alone. As merged, --domain futures cannot run: the provider imports norgatedata and nothing installs it.

These five commits are what the Windows box asked for, and they were written after #7 merged, so they need their own PR rather than being stacked onto merged history.

What the real box found that offline testing could not

Two defects, on the first two contacts, and the pattern matters more than either.

1. --domain futures stopped at the import guard. The provider shipped without its dependency: cotdata declares norgate = ["norgatedata"] and this repo had no such extra. The guard behaved correctly — it refused rather than half-producing — but its message sent the reader to --domain equities, which is right for a Mac or Linux box and exactly wrong for the machine that is supposed to produce futures. A missing package and a Windows-only vendor are two problems with different fixes, and the message now separates them and names the install.

2. get_bars had no volume= parameter. The producer half of volume reconstruction was ported — the provider writes Volume_Reconstructed, FirstVolume, SecondVolume, Volume_Source — and the consumer switch that serves them was not, so cotdata's get_prices(volume="reconstructed") had no counterpart. npf's ml/labels.py:50 passes volume= through, so a repointed call would have raised TypeError.

The docstring carries crowdmon's measurement rather than restating the names, because the names are a trap pointing opposite to intuition: reconstructed sums exactly two expiries and is 0.52 of total volume in natural gas and 0.54 in crude, so front is what a whole-market denominator wants. A test pins reconstructed < front so the docstring cannot drift from it. Equities refuse the parameter outright.

Neither defect is visible to a suite that cannot install the vendor or call a parameter that does not exist.

The verification, and the harness that produced it

scripts/verify_against_cotdata.py compares a marketdata futures store against cotdata's, reading both stores' parquet directly — no network, no imports of either package except the optional --check-propadj. Exit code gates the promotion. Same posture as cotdata's validate_databento_vs_norgate.py: real data, kept out of CI, comparison logic unit tested.

The tolerance is stricter than that neighbour's, and deliberately. It compares two independent vendors whose roll calendars and anchors differ, so only shape can agree. This compares one vendor through two code paths, so the passthrough columns must be exactly equal.

Result on the Windows producer, 2026-08-09:

symbol	rows per tier	passthrough	reconstruction
ES	7,279	identical	identical
CL	10,887	identical	identical
GC	12,156	identical	identical
ZS	12,271	identical	identical
DC	7,299	identical	identical

49,892 rows per tier, both tiers, plus contract specs for all five, exit 0. That is what ADR-0007 §7.5 needs before cotdata's price code is deleted, and it was obtainable only while both halves exist.

The harness reported silence as agreement, which is how defect 2 surfaced

It printed reconstruction columns only when they differed, so "compared and identical" and "never compared" rendered as the same nothing — a PASS could not be distinguished from a PASS that skipped half the frame. It now names every column family it skipped and prints the ones that matched. The first real run printed no reconstruction lines at all, and chasing that silence is what found the missing volume= switch.

One expectation corrected

The reconstruction columns were expected to drift, since each producer reconstructs incrementally over its own store's history and marketdata's was fresh where cotdata's had months of accumulation. They agree exactly: Norgate's historical individual-contract volumes are immutable and the algorithm is identical, so the incremental path converges on what a full recompute produces. That makes --strict-volume usable rather than theoretical.

Also here

MARKETDATA_STORE on Windows. That box now runs two producers and needs both store roots set at once against different roots — new with the futures domain, and documented nowhere. setx rather than set, with the reason (a variable living only in the current prompt is the usual cause of a scheduled task failing where an interactive shell works), --check as the offline confirmation, and an explicit warning against one shared root: each package keeps a manifest.json at its own root and does a read-modify-write on it.

Testing

119 pass (-m "not network"), ruff clean, plus the live comparison above. The harness's own tests cover that a single changed bar in 36,000 is caught and dated, that matching NaNs are agreement rather than difference, that a reconstruction difference does not mask a price difference, and that a typo'd store path exits 2 instead of reading as "nothing differed".

---

Generated by Claude Code