Norgate futures provider (ADR-0007 step 2)

[mspinola]

ADR-0007 makes cotdata CFTC-positioning-only and moves every bar here. This is the provider that has to exist first — written against cotdata's as a reference rather than moved across a seam, since providers/ held only yfinance.py.

Companion record on the cotdata side: mspinola/cotdata#103 (merged) and mspinola/cotdata#104. Nothing is deleted from cotdata and no consumer is repointed yet, so both producers can run side by side.

Verified against cotdata on the Windows box, 2026-08-09

Run with scripts/verify_against_cotdata.py against a cotdata store built by the original producer:

symbol	rows per tier	passthrough	reconstruction
ES	7,279	identical	identical
CL	10,887	identical	identical
GC	12,156	identical	identical
ZS	12,271	identical	identical
DC	7,299	identical	identical

49,892 rows per tier, both tiers, plus contract specs for all five, exit 0. Exact equality, not a tolerance — both producers drive the same Norgate install through two code paths, so a difference would have been a port bug rather than vendor disagreement. Symbols span an index, an energy, a metal, and the two markets whose backadj history goes non-positive.

That is what ADR-0007 §7.5 needs before cotdata's price code is deleted, and it was obtainable only while both halves exist.

The store could not hold the result

bars/<domain>/<source>/<symbol>.parquet gave backadj and unadj the same path. The second write would have silently replaced the first.

The one-frame-per-symbol layout rests on a property equities never violate: corporate actions are dated events the vendor hands over with the bars, so one stored frame reconstructs any tier. Norgate's back-adjustment is roll splicing the vendor performed, and the stitched calendar spread at each roll appears in no other series it publishes — backadj cannot be derived from unadj, or the reverse. Both are stored facts.

Hence a stored-tier filename component (<symbol>_<tier>.parquet) and SCHEMA_VERSION 1 → 2. Equities keep the flat v1 path and read through the same code, so an existing store is extended rather than migrated.

Worth recording that the futures domain was declared but not reachable. On main:

>>> get_bars("ES", "backadj", domain="futures", source="norgate")
ValueError: tier must be one of ('split', 'raw', 'total'), got 'backadj'

check_tier accepted the tier for the futures domain and adjust() then rejected it. The DOMAIN_TIERS["futures"] entry — added so error messages would be right from day one — was a promise about error messages, not a path.

Both tiers or neither

propadj is derived from the two stored frames, and it is the only futures tier whose percent returns are correct. Measured on the cotdata store: 52.3% of ZS's back-adjusted closes and 41.2% of DC's are non-positive, against one such bar for propadj across all 47 symbols (CL, 2020-04-20, because WTI really settled at −37.63 — ratio adjustment preserves the underlying's sign rather than imposing one).

So the rule is enforced three times, and it cannot be lost by editing any one of them:

1. Producer — both tiers fetched and reconstructed before either is written, so a failure on the second leaves nothing on disk.
2. Consumer — propadj against one stored tier raises and names the missing one. Empty would read as "no data for this symbol" when the truth is "the producer half-finished".
3. Test — parametrised over both halves.

Loudness is deliberate: additive back-adjusted percent volatility is ~200x too high for soybeans and 0.47x for gold, and 0.47x never goes negative and passes every implausibility screen a spot check would apply.

What the Windows box found that offline testing could not

Two defects, on the first two contacts, and the pattern matters more than either:

• --domain futures stopped at the import guard. The provider shipped without its dependency: cotdata declares norgate = ["norgatedata"] and this repo had no such extra, so nothing installed it. The guard was right to refuse; its message was wrong for the one machine that matters, sending the Windows producer to --domain equities. Now declared, with the message naming the extra and a test asserting both.
• get_bars had no volume= parameter. The producer half of volume reconstruction was ported and the consumer half was not, so the columns were written and nothing served them. npf's ml/labels.py:50 passes volume= through, so a repointed call would have raised TypeError. The docstring carries crowdmon's measurement rather than restating the names, because they are a trap: reconstructed sums exactly two expiries — 0.52 of total volume in natural gas, 0.54 in crude — so front is the whole-market series. A test pins reconstructed < front.

Neither is visible to a suite that cannot install the vendor or call a parameter that does not exist.

Three things the port broke quietly, all fixed

• provenance() and --pin assumed one series per symbol. provenance("ES") returned None, and an unscoped --pin took manifest keys as symbol names, so futures arrived as the "symbol" ES_backadj. Both are tier-aware now. Snapshot v1 → v2: an entry is one stored series, so --symbols ES pins ES_backadj and ES_unadj — pinning one would leave propadj half covered and let a volatility figure verify green against a store that had moved. v1 snapshots still verify unchanged.
• --check ragged every row — a fixed 10-character symbol column against futures/norgate/ES_backadj. Width now comes from the data.
• The verifier reported silence as agreement. It printed reconstruction columns only when they differed, so "compared and identical" and "never compared" rendered the same. It now names every column family it skipped — which is how the missing volume= switch was found.

Windows-only, and the CLI says so

norgatedata drives a local Norgate Data Updater install, which is Windows-only. --bars skips the futures half with a message on other machines rather than failing the run; --domain futures explains why instead of raising ModuleNotFoundError. The README documents setx for both store roots, since that box now runs two producers against two roots.

Not ported

MME / MFS (MSCI EM and EAFE). Norgate carries no continuous series for either, and cotdata prices them off the EEM and EFA ETF proxies through yfinance. Serving them here needs a futures-domain path in the yfinance provider — separate work from the Norgate producer. They are absent from the futures registry rather than present and unserviceable. Confirmed this costs crowdmon nothing: both already fail its coverage() for the same underlying reason.

Testing

119 pass (-m "not network"), ruff clean, plus the live comparison above. scripts/verify_against_cotdata.py needs two real stores so it cannot run in CI; its comparison logic is unit tested against synthetic frames, including that a single changed bar in 36,000 is caught and dated — same posture as cotdata's validate_databento_vs_norgate.py.