From c3a066aded8d63af1e5743a765d579921e9f999b Mon Sep 17 00:00:00 2001 From: "Matt S." Date: Sun, 9 Aug 2026 01:33:11 +0000 Subject: [PATCH 1/6] Add the port-verification harness, and unit test its comparison logic MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The provider cannot be exercised anywhere but the Windows box, and the one consumer that would have failed loudly on a wrong series — crowdmon, whose strict tier requirements were why the work order put it first — was deprecated with no consumers left. Nothing exercises propadj until the deferred npf pass. So the comparison against cotdata's existing store is the only evidence the port preserved the numbers, and it is available only until ADR-0007 §7.5 deletes the half being compared against. That makes it worth more than two commands and an eyeball. The script reads both stores' parquet directly, no network and no imports of either package, and exits non-zero so it can gate the promotion. Same shape as cotdata's validate_databento_vs_norgate.py, which is the precedent for a real-data harness kept out of CI with its logic unit tested. The tolerance is the part worth noting. That neighbouring harness compares two INDEPENDENT vendors, whose roll calendars and back-adjust anchors differ, so only shape can agree. This compares one vendor through two code paths, so the passthrough columns must be EXACTLY equal and a float away is a port bug. The exception is the volume reconstruction: both producers compute it identically but incrementally over their own store's history, so a fresh marketdata store and a months-old cotdata one legitimately differ there. Those columns are reported rather than failed on, with --strict-volume as the opt-in and a pointer at cotdata's --full. The harness itself is tested here, because a verifier that reports identical on frames that differ is worse than none — it would clear the port for deletion of the only thing it can ever be compared against. 19 tests: a single changed bar is caught and located, matching NaNs are agreement rather than difference, a reconstruction difference does not mask a price difference, MME/MFS are an expected skip rather than a failure, and a typo'd store path exits 2 instead of reading as "nothing differed". Verified end to end on seeded stores: one planted bad bar in 36,000 rows was found and dated, with the reconstruction drift correctly a note. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Jo4iovRfc2fzE9MwcLp7r2 --- scripts/verify_against_cotdata.py | 334 +++++++++++++++++++++++++++ tests/test_verify_against_cotdata.py | 250 ++++++++++++++++++++ 2 files changed, 584 insertions(+) create mode 100644 scripts/verify_against_cotdata.py create mode 100644 tests/test_verify_against_cotdata.py diff --git a/scripts/verify_against_cotdata.py b/scripts/verify_against_cotdata.py new file mode 100644 index 0000000..ece825f --- /dev/null +++ b/scripts/verify_against_cotdata.py @@ -0,0 +1,334 @@ +#!/usr/bin/env python +"""ADR-0007 step 2: prove the futures port preserved the numbers. + +Run this on the Windows box after a `marketdata-update --bars --domain futures`, +while `cotdata`'s price code still exists. Both producers read the same Norgate +install and write separate stores, so the two can be compared directly — until +ADR-0007 §7.5 deletes cotdata's half, at which point this check is gone forever. + +WHY IT MATTERS MORE THAN IT LOOKS. `crowdmon` was the consumer whose tier +requirements were strictest, and the original work order put it first precisely +so it would fail loudly if the provider were wrong. It was deprecated on +2026-08-07 with no consumers left, so that check is gone, and nothing exercises +`propadj` until the deferred `npf` pass. This script is what replaces it. + +WHAT AGREEMENT TO EXPECT — and it is much stricter than the databento harness +next door in cotdata. That one compares two INDEPENDENT vendors, whose roll +calendars and back-adjust anchors legitimately differ, so only the SHAPE can +agree. This compares the same vendor, the same symbol and the same adjustment +through two code paths. The vendor passthrough columns must be **exactly** equal. +A difference is a port bug, not a tolerance question. + +The one column family that may legitimately differ is the volume reconstruction +(`Volume_Reconstructed`, `FirstVolume`, `SecondVolume`, the contract names). +Both producers compute it identically, but incrementally: each only recomputes a +trailing window over what its own store already holds. A fresh marketdata store +recomputes the whole history under today's logic, while cotdata's has accumulated +over months of runs. So those columns are compared and REPORTED, not failed on — +see --strict-volume, and re-run cotdata with `--full` if you want them to match. + +Usage: + python scripts/verify_against_cotdata.py \\ + --cotdata-store "%COTDATA_STORE%" \\ + --marketdata-store "%MARKETDATA_STORE%" \\ + --symbols ES CL GC ZS DC + +Exit code is 0 only if every check passed, so it can gate the promotion. +Reads parquet directly from both stores: no network, and it does not need +`marketdata` or `cotdata` importable except for the optional --check-propadj. +""" +from __future__ import annotations + +import argparse +import sys +from pathlib import Path + +import pandas as pd + +# The columns Norgate hands over and both producers store verbatim. These must be +# identical: both paths call the same `norgatedata.price_timeseries` and rename +# with the same map, so any difference here is the port having changed the data. +PASSTHROUGH = ("Open", "High", "Low", "Close", "Volume", "Open Interest", + "Delivery Month") + +# Computed identically by both, but over each store's own incremental window. +# Compared and reported rather than failed on. See the module docstring. +RECONSTRUCTION = ("Volume_Reconstructed", "FirstVolume", "SecondVolume", + "FirstContract", "SecondContract", "Volume_Source") + +STORED_TIERS = ("backadj", "unadj") + +# Symbols cotdata prices off ETF proxies through yfinance because Norgate carries +# no continuous series. Absent from marketdata's futures registry on purpose, so +# their absence is reported as expected rather than as a missing symbol. +EXPECTED_ABSENT = ("MME", "MFS") + + +def _norm(df: pd.DataFrame) -> pd.DataFrame: + df = df.copy() + df.index = pd.to_datetime(df.index).tz_localize(None).normalize() + df.index.name = "Date" + return df.sort_index() + + +def read_cotdata(store: Path, symbol: str, tier: str) -> pd.DataFrame: + """cotdata layout: prices/_.parquet""" + p = store / "prices" / f"{symbol}_{tier}.parquet" + return _norm(pd.read_parquet(p)) if p.exists() else pd.DataFrame() + + +def read_marketdata(store: Path, symbol: str, tier: str) -> pd.DataFrame: + """marketdata layout: bars/futures/norgate/_.parquet""" + p = store / "bars" / "futures" / "norgate" / f"{symbol}_{tier}.parquet" + return _norm(pd.read_parquet(p)) if p.exists() else pd.DataFrame() + + +def compare_column(a: pd.Series, b: pd.Series) -> dict: + """Exact-equality report for one column over a shared index. + + NaN == NaN counts as equal: a missing Open Interest that is missing in both + stores is agreement, and `!=` would call it a difference on every such row. + """ + both_null = a.isna() & b.isna() + if pd.api.types.is_numeric_dtype(a) and pd.api.types.is_numeric_dtype(b): + diff = (a - b).abs() + unequal = ~(both_null | (diff == 0)) + worst = float(diff[unequal].max()) if unequal.any() else 0.0 + else: + unequal = ~(both_null | (a.astype("object") == b.astype("object"))) + worst = float("nan") if unequal.any() else 0.0 + n = int(unequal.sum()) + return { + "n_differing": n, + "worst_abs_diff": worst, + "first_date": str(a.index[unequal][0].date()) if n else None, + } + + +def compare_tier(cot: pd.DataFrame, mkt: pd.DataFrame) -> dict: + """One stored tier, one symbol. Returns a report dict; `ok` is the verdict on + the passthrough columns only.""" + rep: dict = {"cot_rows": len(cot), "mkt_rows": len(mkt), + "passthrough": {}, "reconstruction": {}, "problems": []} + if cot.empty or mkt.empty: + rep["problems"].append( + "absent from " + ("cotdata" if cot.empty else "marketdata")) + rep["ok"] = False + return rep + + rep["cot_span"] = f"{cot.index.min().date()}..{cot.index.max().date()}" + rep["mkt_span"] = f"{mkt.index.min().date()}..{mkt.index.max().date()}" + + common = cot.index.intersection(mkt.index) + rep["n_common"] = len(common) + # Dates one store has and the other does not. A handful at the tail is just + # one producer having run more recently; a gap in the middle is a real + # difference in what was captured. + rep["cot_only"] = len(cot.index.difference(mkt.index)) + rep["mkt_only"] = len(mkt.index.difference(cot.index)) + if len(common) == 0: + rep["problems"].append("no overlapping dates") + rep["ok"] = False + return rep + + c, m = cot.loc[common], mkt.loc[common] + for col in PASSTHROUGH: + if col not in c.columns and col not in m.columns: + continue + if col not in c.columns or col not in m.columns: + rep["problems"].append( + f"{col}: present in only one store " + f"({'cotdata' if col in c.columns else 'marketdata'})") + continue + r = compare_column(c[col], m[col]) + rep["passthrough"][col] = r + if r["n_differing"]: + rep["problems"].append( + f"{col}: {r['n_differing']} of {len(common)} rows differ " + f"(worst {r['worst_abs_diff']}, first {r['first_date']})") + + for col in RECONSTRUCTION: + if col in c.columns and col in m.columns: + rep["reconstruction"][col] = compare_column(c[col], m[col]) + + rep["ok"] = not rep["problems"] + return rep + + +def compare_specs(cot_store: Path, mkt_store: Path, symbols) -> dict: + """contract_specs is one table keyed by Symbol, so compare row by row.""" + rep: dict = {"problems": [], "compared": 0} + cp = cot_store / "metadata" / "contract_specs.parquet" + mp = mkt_store / "metadata" / "contract_specs.parquet" + if not cp.exists() or not mp.exists(): + rep["problems"].append( + f"contract_specs missing in {'cotdata' if not cp.exists() else 'marketdata'} " + f"(run marketdata-update --metadata)") + rep["ok"] = False + return rep + + c = pd.read_parquet(cp).set_index("Symbol") + m = pd.read_parquet(mp).set_index("Symbol") + for sym in symbols: + if sym in EXPECTED_ABSENT: + continue + if sym not in c.index: + continue # cotdata never had it; not a port question + if sym not in m.index: + rep["problems"].append(f"{sym}: no contract_specs row in marketdata") + continue + rep["compared"] += 1 + for col in ("Point Value", "Tick Size", "Tick Value", "Currency", + "Exchange", "Name"): + if col not in c.columns or col not in m.columns: + continue + cv, mv = c.loc[sym, col], m.loc[sym, col] + if pd.isna(cv) and pd.isna(mv): + continue + if cv != mv: + rep["problems"].append(f"{sym}.{col}: cotdata {cv!r}, marketdata {mv!r}") + rep["ok"] = not rep["problems"] + return rep + + +def check_propadj(symbol: str, cot_store: Path) -> dict: + """Run both ratio-adjust implementations over the SAME input frames. + + Store equality is checked separately, so feeding both implementations one pair + of frames isolates the ALGORITHM port from the data. Needs both packages + importable; skipped otherwise. + """ + rep: dict = {"skipped": None, "problems": []} + try: + from cotdata.prices import _ratio_adjust as cot_ratio + + from marketdata.adjust import ratio_adjust as mkt_ratio + except ImportError as e: + rep["skipped"] = f"{e} (needs both packages importable)" + return rep + + unadj = read_cotdata(cot_store, symbol, "unadj") + backadj = read_cotdata(cot_store, symbol, "backadj") + if unadj.empty or backadj.empty: + rep["skipped"] = "cotdata store lacks both tiers for this symbol" + return rep + + mine = mkt_ratio(unadj, backadj) + # cotdata's reads the store itself and keys off COTDATA_STORE, so point it at + # the same store rather than reimplementing its read. + theirs = cot_ratio(symbol) + if theirs.empty: + rep["skipped"] = "cotdata's _ratio_adjust returned empty (is COTDATA_STORE set?)" + return rep + + common = mine.index.intersection(theirs.index) + rep["n_common"] = len(common) + if not len(common): + rep["problems"].append("no overlapping dates between the two propadj series") + rep["ok"] = False + return rep + for col in ("Open", "High", "Low", "Close"): + if col in mine.columns and col in theirs.columns: + d = (mine.loc[common, col] - theirs.loc[common, col]).abs().max() + rep[col] = float(d) + # Same algorithm on the same inputs: floating-point noise only. + if pd.notna(d) and d > 1e-9: + rep["problems"].append(f"{col}: max abs diff {d:g} exceeds 1e-9") + rep["ok"] = not rep["problems"] + return rep + + +def main(argv=None) -> int: + p = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + p.add_argument("--cotdata-store", required=True) + p.add_argument("--marketdata-store", required=True) + p.add_argument("--symbols", nargs="+", default=["ES", "CL", "GC", "ZS", "DC"], + help="default set spans an index, an energy, a metal, and the two " + "markets whose backadj history goes non-positive") + p.add_argument("--strict-volume", action="store_true", + help="fail on reconstructed-volume differences too. Only meaningful " + "after a cotdata run with --full, since both producers " + "reconstruct incrementally over their own store's history") + p.add_argument("--check-propadj", action="store_true", + help="also run cotdata's and marketdata's ratio-adjust over the same " + "frames. Needs both packages importable") + args = p.parse_args(argv) + + cot_store, mkt_store = Path(args.cotdata_store), Path(args.marketdata_store) + for label, path in (("cotdata", cot_store), ("marketdata", mkt_store)): + if not path.exists(): + print(f"ERROR: {label} store does not exist: {path}") + return 2 + + failures = [] + print(f"cotdata store {cot_store}") + print(f"marketdata store {mkt_store}\n") + + for sym in args.symbols: + if sym in EXPECTED_ABSENT: + print(f"{sym}: skipped — Norgate carries no continuous series, priced off " + f"an ETF proxy in cotdata and deliberately not ported") + continue + print(f"── {sym} " + "─" * 60) + + for tier in STORED_TIERS: + rep = compare_tier(read_cotdata(cot_store, sym, tier), + read_marketdata(mkt_store, sym, tier)) + status = "OK " if rep["ok"] else "FAIL" + head = (f" {status} {tier:8s} cot {rep['cot_rows']:6d} rows, " + f"mkt {rep['mkt_rows']:6d}") + if "n_common" in rep: + head += (f", {rep['n_common']} common" + + (f", {rep['cot_only']} cot-only" if rep["cot_only"] else "") + + (f", {rep['mkt_only']} mkt-only" if rep["mkt_only"] else "")) + print(head) + for prob in rep["problems"]: + print(f" ! {prob}") + if not rep["ok"]: + failures.append(f"{sym}/{tier}") + + for col, r in rep["reconstruction"].items(): + if r["n_differing"]: + note = ("FAIL" if args.strict_volume else "note") + print(f" {note} {col}: {r['n_differing']} rows differ " + f"(first {r['first_date']}) — expected unless cotdata was " + f"last run with --full") + if args.strict_volume: + failures.append(f"{sym}/{tier}/{col}") + + if args.check_propadj: + r = check_propadj(sym, cot_store) + if r.get("skipped"): + print(f" SKIP propadj {r['skipped']}") + else: + print(f" {'OK ' if r['ok'] else 'FAIL'} propadj " + f"{r.get('n_common', 0)} common rows, " + f"max close diff {r.get('Close', float('nan')):g}") + for prob in r["problems"]: + print(f" ! {prob}") + if not r["ok"]: + failures.append(f"{sym}/propadj") + + print("\n── contract specs " + "─" * 51) + specs = compare_specs(cot_store, mkt_store, args.symbols) + print(f" {'OK ' if specs['ok'] else 'FAIL'} {specs['compared']} symbols compared") + for prob in specs["problems"]: + print(f" ! {prob}") + if not specs["ok"]: + failures.append("contract_specs") + + print() + if failures: + print(f"FAILED: {len(failures)} check(s) — {', '.join(failures)}") + print("\nThe two producers read the same Norgate install, so the passthrough " + "columns differing means the port changed the data. Do not promote.") + return 1 + print("PASSED: every compared series is identical between the two stores.") + print("\nThat is the evidence ADR-0007 §7.5 needs before cotdata's price code is " + "deleted, and it is only obtainable while both halves still exist.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/test_verify_against_cotdata.py b/tests/test_verify_against_cotdata.py new file mode 100644 index 0000000..049bd09 --- /dev/null +++ b/tests/test_verify_against_cotdata.py @@ -0,0 +1,250 @@ +"""The port-verification harness, tested against synthetic frames. + +The script itself needs two real stores and a Norgate install, so it cannot run in +CI. Its comparison logic can, and must: a verifier that reports "identical" on +frames that differ is worse than no verifier, because it would clear the port for +deletion of the only thing it could ever be compared against. + +Same posture as cotdata's `tests/test_validate_databento.py` beside the harness it +covers. +""" +import importlib.util +import sys +from pathlib import Path + +import pandas as pd +import pytest + +_SCRIPT = Path(__file__).resolve().parents[1] / "scripts" / "verify_against_cotdata.py" + + +@pytest.fixture(scope="module") +def verify(): + spec = importlib.util.spec_from_file_location("verify_against_cotdata", _SCRIPT) + mod = importlib.util.module_from_spec(spec) + sys.modules[spec.name] = mod + spec.loader.exec_module(mod) + return mod + + +def frame(closes, *, volume=100.0, delivery=None, start="2020-01-01"): + idx = pd.date_range(start, periods=len(closes), freq="D") + df = pd.DataFrame({"Open": closes, "High": closes, "Low": closes, + "Close": closes, "Volume": volume, + "Open Interest": 1000.0}, index=idx) + if delivery is not None: + df["Delivery Month"] = delivery + df.index.name = "Date" + return df + + +# ── The verdict has to be right in both directions ──────────────────────── +def test_identical_frames_pass(verify): + df = frame([1.0, 2.0, 3.0], delivery=["A", "A", "B"]) + rep = verify.compare_tier(df, df.copy()) + assert rep["ok"] + assert rep["problems"] == [] + assert rep["n_common"] == 3 + + +def test_a_single_changed_price_fails_and_is_located(verify): + """The check that earns the harness its keep. One wrong bar in ten thousand is + exactly what a silent port bug looks like.""" + cot = frame([1.0, 2.0, 3.0]) + mkt = cot.copy() + mkt.iloc[1, mkt.columns.get_loc("Close")] = 2.0001 + + rep = verify.compare_tier(cot, mkt) + assert not rep["ok"] + assert rep["passthrough"]["Close"]["n_differing"] == 1 + assert rep["passthrough"]["Close"]["first_date"] == "2020-01-02" + assert any("Close" in p for p in rep["problems"]) + + +def test_tolerance_is_exact_not_approximate(verify): + """Two INDEPENDENT vendors would need a tolerance. These are two code paths + over one vendor, so a float away is still a difference.""" + cot = frame([100.0]) + mkt = frame([100.0 + 1e-12]) + assert not verify.compare_tier(cot, mkt)["ok"] + + +def test_a_changed_delivery_month_fails(verify): + """Non-numeric, and it drives roll detection — so propadj is wrong downstream + of a difference here, silently.""" + cot = frame([1.0, 2.0], delivery=["2020H", "2020H"]) + mkt = frame([1.0, 2.0], delivery=["2020H", "2020M"]) + rep = verify.compare_tier(cot, mkt) + assert not rep["ok"] + assert rep["passthrough"]["Delivery Month"]["n_differing"] == 1 + + +def test_matching_nans_are_agreement_not_difference(verify): + """An Open Interest Norgate never supplied is missing in both stores. Comparing + with `!=` would call every such row a difference and drown the real ones.""" + cot = frame([1.0, 2.0]) + cot["Open Interest"] = [float("nan"), 5.0] + rep = verify.compare_tier(cot, cot.copy()) + assert rep["ok"] + assert rep["passthrough"]["Open Interest"]["n_differing"] == 0 + + +def test_a_nan_on_one_side_only_is_a_difference(verify): + cot = frame([1.0, 2.0]) + mkt = cot.copy() + mkt["Open Interest"] = [float("nan"), 1000.0] + assert not verify.compare_tier(cot, mkt)["ok"] + + +# ── Absence and partial overlap ─────────────────────────────────────────── +def test_missing_from_marketdata_fails_and_says_which_side(verify): + rep = verify.compare_tier(frame([1.0]), pd.DataFrame()) + assert not rep["ok"] + assert "marketdata" in rep["problems"][0] + + +def test_missing_from_cotdata_fails_and_says_which_side(verify): + rep = verify.compare_tier(pd.DataFrame(), frame([1.0])) + assert not rep["ok"] + assert "cotdata" in rep["problems"][0] + + +def test_differing_tails_are_counted_not_failed(verify): + """One producer having run more recently is normal. Only the OVERLAP is + compared, and the extra days are reported so a gap in the middle is still + visible.""" + cot = frame([1.0, 2.0, 3.0, 4.0]) + mkt = frame([1.0, 2.0, 3.0]) + rep = verify.compare_tier(cot, mkt) + assert rep["ok"] + assert rep["n_common"] == 3 + assert rep["cot_only"] == 1 + assert rep["mkt_only"] == 0 + + +def test_no_overlap_at_all_fails(verify): + cot = frame([1.0, 2.0], start="2020-01-01") + mkt = frame([1.0, 2.0], start="2021-01-01") + rep = verify.compare_tier(cot, mkt) + assert not rep["ok"] + assert "no overlapping dates" in rep["problems"][0] + + +# ── Reconstruction columns are reported, never failed ───────────────────── +def test_reconstructed_volume_differences_do_not_fail_the_tier(verify): + """Both producers reconstruct incrementally over their own store's history, so + a fresh marketdata store and a months-old cotdata one legitimately differ here. + Reported for the operator; --strict-volume is the opt-in.""" + cot = frame([1.0, 2.0]) + cot["Volume_Reconstructed"] = [10.0, 20.0] + mkt = frame([1.0, 2.0]) + mkt["Volume_Reconstructed"] = [10.0, 999.0] + + rep = verify.compare_tier(cot, mkt) + assert rep["ok"], "a reconstruction difference must not fail the passthrough verdict" + assert rep["reconstruction"]["Volume_Reconstructed"]["n_differing"] == 1 + + +def test_a_price_difference_still_fails_alongside_a_reconstruction_difference(verify): + """The reconstruction leniency must not swallow a real one.""" + cot = frame([1.0, 2.0]) + cot["Volume_Reconstructed"] = [10.0, 20.0] + mkt = frame([1.0, 2.5]) + mkt["Volume_Reconstructed"] = [10.0, 999.0] + assert not verify.compare_tier(cot, mkt)["ok"] + + +# ── Store readers match each layout ─────────────────────────────────────── +def test_readers_use_each_stores_own_layout(verify, tmp_path): + """The two stores name files differently, and reading the wrong path would + report a symbol absent rather than compare it.""" + cot_store, mkt_store = tmp_path / "cot", tmp_path / "mkt" + (cot_store / "prices").mkdir(parents=True) + (mkt_store / "bars" / "futures" / "norgate").mkdir(parents=True) + + frame([1.0, 2.0]).to_parquet(cot_store / "prices" / "ES_backadj.parquet") + frame([1.0, 2.0]).to_parquet( + mkt_store / "bars" / "futures" / "norgate" / "ES_backadj.parquet") + + assert len(verify.read_cotdata(cot_store, "ES", "backadj")) == 2 + assert len(verify.read_marketdata(mkt_store, "ES", "backadj")) == 2 + assert verify.read_cotdata(cot_store, "ES", "unadj").empty + assert verify.read_marketdata(mkt_store, "GC", "backadj").empty + + +# ── Contract specs ──────────────────────────────────────────────────────── +def _write_specs(root, rows): + (root / "metadata").mkdir(parents=True, exist_ok=True) + pd.DataFrame(rows).to_parquet(root / "metadata" / "contract_specs.parquet") + + +def test_matching_specs_pass_and_a_changed_point_value_fails(verify, tmp_path): + """Point Value is the multiplier every notional and risk-unit figure scales by, + so a silent change here rescales an entire book.""" + cot, mkt = tmp_path / "cot", tmp_path / "mkt" + rows = [{"Symbol": "ES", "Point Value": 50.0, "Tick Size": 0.25, + "Currency": "USD"}] + _write_specs(cot, rows) + _write_specs(mkt, rows) + assert verify.compare_specs(cot, mkt, ["ES"])["ok"] + + _write_specs(mkt, [{"Symbol": "ES", "Point Value": 5.0, "Tick Size": 0.25, + "Currency": "USD"}]) + rep = verify.compare_specs(cot, mkt, ["ES"]) + assert not rep["ok"] + assert "Point Value" in rep["problems"][0] + + +def test_a_symbol_missing_from_marketdata_specs_fails(verify, tmp_path): + cot, mkt = tmp_path / "cot", tmp_path / "mkt" + _write_specs(cot, [{"Symbol": "ES", "Point Value": 50.0}]) + _write_specs(mkt, [{"Symbol": "GC", "Point Value": 100.0}]) + rep = verify.compare_specs(cot, mkt, ["ES"]) + assert not rep["ok"] + assert "no contract_specs row in marketdata" in rep["problems"][0] + + +def test_expected_absent_symbols_are_not_reported_as_missing(verify, tmp_path): + """MME/MFS have no Norgate series and are deliberately unported. Failing on + them would train the operator to ignore a red run.""" + cot, mkt = tmp_path / "cot", tmp_path / "mkt" + _write_specs(cot, [{"Symbol": "MME", "Point Value": 50.0}]) + _write_specs(mkt, [{"Symbol": "ES", "Point Value": 50.0}]) + assert verify.compare_specs(cot, mkt, ["MME"])["ok"] + assert "MME" in verify.EXPECTED_ABSENT + + +def test_missing_specs_table_is_a_failure_naming_the_fix(verify, tmp_path): + cot, mkt = tmp_path / "cot", tmp_path / "mkt" + _write_specs(cot, [{"Symbol": "ES", "Point Value": 50.0}]) + mkt.mkdir(parents=True) + rep = verify.compare_specs(cot, mkt, ["ES"]) + assert not rep["ok"] + assert "--metadata" in rep["problems"][0] + + +# ── Exit code is the gate ───────────────────────────────────────────────── +def test_exit_code_is_zero_only_when_everything_matches(verify, tmp_path): + cot, mkt = tmp_path / "cot", tmp_path / "mkt" + (cot / "prices").mkdir(parents=True) + bars = mkt / "bars" / "futures" / "norgate" + bars.mkdir(parents=True) + for tier in ("backadj", "unadj"): + frame([1.0, 2.0]).to_parquet(cot / "prices" / f"ES_{tier}.parquet") + frame([1.0, 2.0]).to_parquet(bars / f"ES_{tier}.parquet") + specs = [{"Symbol": "ES", "Point Value": 50.0}] + _write_specs(cot, specs) + _write_specs(mkt, specs) + + argv = ["--cotdata-store", str(cot), "--marketdata-store", str(mkt), + "--symbols", "ES"] + assert verify.main(argv) == 0 + + frame([1.0, 99.0]).to_parquet(bars / "ES_unadj.parquet") + assert verify.main(argv) == 1 + + +def test_a_nonexistent_store_exits_two_rather_than_passing(verify, tmp_path): + """A typo'd path must not read as 'nothing differed'.""" + assert verify.main(["--cotdata-store", str(tmp_path / "nope"), + "--marketdata-store", str(tmp_path)]) == 2 From a710f9b64e4d249b224ac0e3bd1eb69b4bcced4f Mon Sep 17 00:00:00 2001 From: "Matt S." Date: Sun, 9 Aug 2026 01:41:28 +0000 Subject: [PATCH 2/6] Declare the norgate extra, which the provider shipped without MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `--domain futures` stopped at the import guard on the Windows producer because nothing installs norgatedata: the provider landed with a dependency no extra declares. cotdata has carried `norgate = ["norgatedata"]` all along and the port did not bring it across. The guard behaved correctly — it refused rather than half-producing — but its message sent the reader to --domain equities, which is right for a Mac or Linux box and exactly wrong for the machine that is supposed to produce futures. A missing package and a Windows-only vendor are two problems with two different fixes, and the message now separates them and names the install. Two tests, because the guard being right is what hid this. One asserts the extra exists and carries norgatedata, so a provider whose vendor package nothing installs cannot ship again. The other asserts the message names both the extra and the platform constraint, so it cannot quietly regress to advice that only suits the machines which were never going to run it. README gains the Windows producer install line for the same reason. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Jo4iovRfc2fzE9MwcLp7r2 --- README.md | 11 ++++++++ pyproject.toml | 7 +++++ src/marketdata/providers/norgate.py | 10 +++++-- tests/test_futures.py | 44 +++++++++++++++++++++++++++++ 4 files changed, 69 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 14124e1..dd3f6cf 100644 --- a/README.md +++ b/README.md @@ -23,6 +23,17 @@ moves every bar here. uv venv --python 3.11 && uv pip install -e ".[yahoo,dev]" "setuptools<81" ``` +On the **Windows futures producer**, add the `norgate` extra — nothing else pulls +`norgatedata`, and without it `--domain futures` stops before it fetches: + +```bash +uv pip install -e ".[yahoo,norgate,dev]" "setuptools<81" +``` + +Installing it elsewhere does not help. It drives a locally installed Norgate Data +Updater rather than an API, and NDU is Windows-only, so every other machine reads +a synced store instead of producing one. + ## Use ```bash diff --git a/pyproject.toml b/pyproject.toml index a0f515a..b3c9a6b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -32,6 +32,13 @@ Issues = "https://github.com/mspinola/marketdata/issues" # history(auto_adjust=...) has changed across yfinance versions — see # docs/design.md. Verified against 1.5.2. yahoo = ["yfinance>=1.5.2"] +# The futures producer, WINDOWS ONLY. The package installs anywhere, but it drives +# a locally installed Norgate Data Updater rather than an API, and NDU exists for +# Windows alone — so on a Mac or Linux box it imports fine and then has nothing to +# talk to. That is why this is an extra rather than a dependency, and why the +# provider guards the import and the service separately: a missing package and a +# stopped NDU are different problems with different fixes. +norgate = ["norgatedata"] dev = ["pytest>=7", "ruff==0.15.22"] [project.scripts] diff --git a/src/marketdata/providers/norgate.py b/src/marketdata/providers/norgate.py index b77ec30..c5497f6 100644 --- a/src/marketdata/providers/norgate.py +++ b/src/marketdata/providers/norgate.py @@ -87,9 +87,13 @@ def _require_norgate_service() -> None: # the normal state of every other machine. raise RuntimeError( "norgatedata is not installed, so this machine cannot produce futures " - "bars. It drives a local Norgate Data Updater install, which is " - "Windows-only — a Mac or Linux box reads a SYNCED store instead of " - "producing one. Use --domain equities here.") from e + "bars.\n" + " On the Windows producer, this is just the missing extra:\n" + " uv pip install -e \".[norgate]\"\n" + " Anywhere else, installing it will not help: it drives a local " + "Norgate Data Updater install and NDU is Windows-only, so a Mac or " + "Linux box reads a SYNCED store rather than producing one. Use " + "--domain equities there.") from e try: reachable = bool(norgatedata.status()) except BaseException: # noqa: BLE001 — never let the probe itself take us down diff --git a/tests/test_futures.py b/tests/test_futures.py index 43f7b0b..6c39388 100644 --- a/tests/test_futures.py +++ b/tests/test_futures.py @@ -412,3 +412,47 @@ def test_provenance_reports_each_futures_tier_separately(tmp_store): assert provenance("ES").n_rows == 3 assert provenance("ES", tier="unadj").n_rows == 2 assert "backadj" in provenance("ES").describe() + + +# ── Packaging ───────────────────────────────────────────────────────────── +def test_the_norgate_extra_is_declared(): + """A provider whose vendor package nothing installs is a provider nobody can run. + + This shipped missing: the futures provider landed with no `norgate` extra, so + `--domain futures` on the Windows producer stopped at the import guard with the + package never having been pulled. The guard did its job; the packaging had not. + """ + from pathlib import Path + + import tomllib + + pyproject = Path(__file__).resolve().parents[1] / "pyproject.toml" + extras = tomllib.loads(pyproject.read_text())["project"]["optional-dependencies"] + assert "norgate" in extras, ( + "providers/norgate.py imports norgatedata, but no extra installs it") + assert any("norgatedata" in dep for dep in extras["norgate"]) + + +def test_the_missing_package_message_names_the_extra_that_fixes_it(): + """Two different problems with two different fixes: a missing package is an + install away on Windows, and unfixable anywhere else. The message has to + separate them or it sends the Windows producer to --domain equities.""" + import builtins + + real_import = builtins.__import__ + + def no_norgatedata(name, *a, **kw): + if name == "norgatedata": + raise ImportError("No module named 'norgatedata'") + return real_import(name, *a, **kw) + + builtins.__import__ = no_norgatedata + try: + with pytest.raises(RuntimeError) as e: + nprov._require_norgate_service() + finally: + builtins.__import__ = real_import + + msg = str(e.value) + assert "[norgate]" in msg # the fix, for the box that can be fixed + assert "Windows" in msg # and why it is not the fix anywhere else From 93860181e2670931915387ed17984512c7328d2c Mon Sep 17 00:00:00 2001 From: "Matt S." Date: Sun, 9 Aug 2026 01:48:24 +0000 Subject: [PATCH 3/6] Document MARKETDATA_STORE on the Windows box, where two producers now meet MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The variable was documented as required, and config.py refuses by name when it is unset, but every example was `export ...` — bash, on a repo whose only producer machine runs Windows. cotdata carries a whole Windows setup guide with `set`, System Settings, .env and an `echo %VAR%` check; marketdata carried nothing. The larger gap is newer than the port. That box now runs BOTH producers, so it needs COTDATA_STORE and MARKETDATA_STORE set at the same time against different roots, and nothing said so. Until ADR-0007 moved bars here, COTDATA_STORE alone was the whole story. So: setx rather than set, with the reason spelled out, since a variable that lives only in the current prompt is the usual cause of a scheduled task failing where an interactive shell works. Then --check as the cheap confirmation, being manifest-only and offline. And an explicit warning against one shared root: sharing a parent folder is fine and makes the pair easy to sync, but each package keeps a manifest.json at its own root and does a read-modify-write on it, so one root means the two producers eventually drop each other's entries. Python, venv and Task Scheduler setup are identical to cotdata's and are pointed at rather than duplicated. Only the norgate extra and these two variables are marketdata-specific. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Jo4iovRfc2fzE9MwcLp7r2 --- README.md | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/README.md b/README.md index dd3f6cf..d418045 100644 --- a/README.md +++ b/README.md @@ -162,6 +162,44 @@ silently substituting a vendor is what ADR-0006 forbids. The store root may share a parent folder with cotdata's, but the two must not share a `manifest.json`. Both producers do a read-modify-write on it. +### On the Windows futures producer + +That box now runs **two producers**, so it needs **both** store variables set at +once, pointing at **different roots**. This is new with the futures domain: until +ADR-0007 moved bars here, `COTDATA_STORE` alone was the whole story. + +```cmd +setx COTDATA_STORE C:\Users\YourUsername\cotdata_store +setx MARKETDATA_STORE C:\Users\YourUsername\marketdata_store +``` + +`setx` persists; plain `set` lasts only for the current Command Prompt, which is +the usual reason a scheduled task cannot find a store an interactive shell could. +Open a NEW prompt afterwards — `setx` does not affect the one you typed it in — +and verify: + +```cmd +echo %COTDATA_STORE% +echo %MARKETDATA_STORE% +marketdata-update --check +``` + +`--check` reads the manifest and no network, so it is the cheap confirmation that +the variable points where you think. An unset variable is refused by name rather +than defaulted, because a silent default would write a second store somewhere +nobody looks. + +**Do not point them at one root.** Sharing a parent folder is fine and makes the +pair easy to sync; sharing a root is not, because both packages keep a +`manifest.json` at their root and each does a read-modify-write on it, so the two +producers would eventually drop each other's entries. + +Python, virtualenv and Task Scheduler setup are identical to cotdata's and are +not duplicated here — see +[cotdata's Windows setup guide](https://github.com/mspinola/cotdata/blob/main/docs/WINDOWS_SETUP.md). +The only marketdata-specific pieces are the `norgate` extra in **Install** above +and the two variables here. + ## Tests ```bash From c06c7d83aae6164e57b92b13dc5771446932289c Mon Sep 17 00:00:00 2001 From: "Matt S." Date: Sun, 9 Aug 2026 01:59:06 +0000 Subject: [PATCH 4/6] Port the volume= consumer switch, and stop the verifier reporting silence as agreement MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first real run passed on every passthrough column across 49,892 rows and five symbols, and printed nothing at all about the reconstruction columns. That was predicted to print notes, so the silence was the finding. Two causes, both mine. The verifier reported reconstruction columns only when they DIFFERED, so "compared and identical" and "never compared" rendered as the same nothing. A PASS therefore could not be read as "everything was compared", which is the one thing this harness exists to establish before ADR-0007 §7.5 deletes the only store it can ever compare against. It now names every column family it skipped and why, and prints the ones that matched. And the reason there may have been nothing to compare: get_bars never grew the volume= parameter. The producer half was ported — _reconstruct_volume writes Volume_Reconstructed, FirstVolume, SecondVolume and Volume_Source — but the consumer switch that serves them was not, so cotdata's get_prices(volume="reconstructed") had no counterpart here. npf's ml/labels.py passes volume= through, so a repointed call would have raised TypeError. The docstring carries crowdmon's measurement rather than restating the names, because the names are a trap that points the opposite way to intuition: `reconstructed` sums exactly two expiries and is 0.52 of total volume in natural gas and 0.54 in crude, so `front` is what a whole-market denominator wants. A test pins that reconstructed < front so the docstring cannot drift from it. Equities refuse the parameter outright, since summing two expiries is not a thing an equity has. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Jo4iovRfc2fzE9MwcLp7r2 --- scripts/verify_against_cotdata.py | 29 ++++++++++-- src/marketdata/bars.py | 50 +++++++++++++++++++++ tests/test_futures.py | 67 ++++++++++++++++++++++++++++ tests/test_verify_against_cotdata.py | 34 ++++++++++++++ 4 files changed, 177 insertions(+), 3 deletions(-) diff --git a/scripts/verify_against_cotdata.py b/scripts/verify_against_cotdata.py index ece825f..b7ca2f1 100644 --- a/scripts/verify_against_cotdata.py +++ b/scripts/verify_against_cotdata.py @@ -109,7 +109,8 @@ def compare_tier(cot: pd.DataFrame, mkt: pd.DataFrame) -> dict: """One stored tier, one symbol. Returns a report dict; `ok` is the verdict on the passthrough columns only.""" rep: dict = {"cot_rows": len(cot), "mkt_rows": len(mkt), - "passthrough": {}, "reconstruction": {}, "problems": []} + "passthrough": {}, "reconstruction": {}, "problems": [], + "not_compared": []} if cot.empty or mkt.empty: rep["problems"].append( "absent from " + ("cotdata" if cot.empty else "marketdata")) @@ -147,9 +148,20 @@ def compare_tier(cot: pd.DataFrame, mkt: pd.DataFrame) -> dict: f"{col}: {r['n_differing']} of {len(common)} rows differ " f"(worst {r['worst_abs_diff']}, first {r['first_date']})") + # Coverage is reported, not just differences. A column absent from one store + # is skipped, and if that skip were silent it would look exactly like a + # column that matched — so a PASS could not be read as "everything was + # compared". Naming the untested columns is the difference between evidence + # and a green light. for col in RECONSTRUCTION: - if col in c.columns and col in m.columns: + in_c, in_m = col in c.columns, col in m.columns + if in_c and in_m: rep["reconstruction"][col] = compare_column(c[col], m[col]) + elif in_c or in_m: + rep["not_compared"].append( + f"{col} (only in {'cotdata' if in_c else 'marketdata'})") + else: + rep["not_compared"].append(f"{col} (in neither store)") rep["ok"] = not rep["problems"] return rep @@ -261,7 +273,7 @@ def main(argv=None) -> int: print(f"ERROR: {label} store does not exist: {path}") return 2 - failures = [] + failures, untested = [], [] print(f"cotdata store {cot_store}") print(f"marketdata store {mkt_store}\n") @@ -288,6 +300,10 @@ def main(argv=None) -> int: if not rep["ok"]: failures.append(f"{sym}/{tier}") + matched = [c for c, r in rep["reconstruction"].items() + if not r["n_differing"]] + if matched: + print(f" ok reconstruction identical: {', '.join(matched)}") for col, r in rep["reconstruction"].items(): if r["n_differing"]: note = ("FAIL" if args.strict_volume else "note") @@ -296,6 +312,9 @@ def main(argv=None) -> int: f"last run with --full") if args.strict_volume: failures.append(f"{sym}/{tier}/{col}") + if rep["not_compared"]: + print(f" ?? NOT COMPARED: {'; '.join(rep['not_compared'])}") + untested.append(f"{sym}/{tier}") if args.check_propadj: r = check_propadj(sym, cot_store) @@ -319,6 +338,10 @@ def main(argv=None) -> int: failures.append("contract_specs") print() + if untested: + print(f"NOTE: {len(untested)} tier(s) had columns present in only one store, or " + f"in neither, and those columns were NOT compared — see '?? NOT COMPARED' " + f"above. The verdict below covers what was compared.") if failures: print(f"FAILED: {len(failures)} check(s) — {', '.join(failures)}") print("\nThe two producers read the same Norgate install, so the passthrough " diff --git a/src/marketdata/bars.py b/src/marketdata/bars.py index bf6cf50..af49e04 100644 --- a/src/marketdata/bars.py +++ b/src/marketdata/bars.py @@ -32,6 +32,7 @@ def default_source_for(symbol: str) -> str: def get_bars(symbol: str, adjustment: Optional[str] = None, *, source: Optional[str] = None, domain: Optional[str] = None, start: Optional[str] = None, end: Optional[str] = None, + volume: str = "front", include_capital_gains: bool = False) -> pd.DataFrame: """Daily bars for `symbol`, adjusted to `adjustment`. @@ -56,6 +57,23 @@ def get_bars(symbol: str, adjustment: Optional[str] = None, *, where additive adjustment has driven the back-history through zero (ZS, DC). See `adjust.ratio_adjust`. + `volume` selects which series the `Volume` column carries (futures only): + 'front' : continuous front-month volume as Norgate reports it. + Default. + 'reconstructed' : FirstVolume + SecondVolume — the two highest-volume + expiries trading that day, with a per-row fall-back to + front-month where individual contracts are unavailable. + Adds a `Volume_Source` column ('reconstructed' / 'raw') + so a consumer can tell the fall-back rows apart. + + **Read those two again: the names are a trap, and it points the opposite way + to intuition.** `reconstructed` is NOT whole-market. It sums exactly two + expiries, which is 0.52 of total volume in natural gas and 0.54 in crude, so + it understates most in precisely the markets with the deepest curves. + `front` is the series to divide a whole-market quantity by. Measured in + crowdmon, whose `futures/volume.py` refuses anything but `front` for that + reason; carried here because the naming will mislead the next reader too. + `domain` is resolved from the registry and rarely passed. `source` pins the vendor. Omit it and the registry resolves one for this deployment. Pass it explicitly to compare vendors on the same symbol, which is the point of @@ -71,6 +89,13 @@ def get_bars(symbol: str, adjustment: Optional[str] = None, *, check_tier(adjustment, dom) src = source or default_source_for(symbol) + if volume not in ("front", "reconstructed"): + raise ValueError(f"volume must be 'front' or 'reconstructed', got {volume!r}") + if volume == "reconstructed" and dom != "futures": + raise ValueError( + f"volume='reconstructed' is a futures concept (it sums the two " + f"highest-volume expiries) and {symbol!r} is in domain {dom!r}.") + if dom == "futures": out = _futures_bars(symbol, src, adjustment) else: @@ -78,6 +103,8 @@ def get_bars(symbol: str, adjustment: Optional[str] = None, *, include_capital_gains=include_capital_gains) if out.empty: return out + if volume == "reconstructed": + out = _reconstructed_volume(out) if start is not None: out = out[out.index >= pd.Timestamp(start)] @@ -106,6 +133,29 @@ def _missing(symbol: str, dom: str, src: str, tier=None) -> None: f"MARKETDATA_PRICE_SOURCE. Vendors are never silently substituted.") +def _reconstructed_volume(df: pd.DataFrame) -> pd.DataFrame: + """Swap `Volume` for the reconstructed series, with a per-row fall-back. + + The producer already writes `Volume_Reconstructed == Volume` on rows it could + not reconstruct, so reading the column is fall-back-safe where it exists. The + guards are for a store written before reconstruction, or a stray NaN: both + degrade to front-month volume and SAY SO in `Volume_Source`, because a + consumer comparing volume across symbols has to be able to exclude the rows + that are not really reconstructed. + """ + out = df.copy() + if "Volume_Reconstructed" in out.columns: + rec = out["Volume_Reconstructed"] + out["Volume"] = rec.where(rec.notna(), out["Volume"]) + if "Volume_Source" not in out.columns: + out["Volume_Source"] = "reconstructed" + out.loc[rec.isna(), "Volume_Source"] = "raw" + else: + # Store predates reconstruction, so every row is front-month. + out["Volume_Source"] = "raw" + return out + + def _equity_bars(symbol: str, dom: str, src: str, tier: str, *, include_capital_gains: bool) -> pd.DataFrame: """One stored frame, every tier derived from it.""" diff --git a/tests/test_futures.py b/tests/test_futures.py index 6c39388..1c04925 100644 --- a/tests/test_futures.py +++ b/tests/test_futures.py @@ -456,3 +456,70 @@ def no_norgatedata(name, *a, **kw): msg = str(e.value) assert "[norgate]" in msg # the fix, for the box that can be fixed assert "Windows" in msg # and why it is not the fix anywhere else + + +# ── Reconstructed volume, consumer side ─────────────────────────────────── +def _recon_frame(): + df = frame([1.0, 2.0, 3.0]) + df["Volume"] = [100.0, 200.0, 300.0] + df["Volume_Reconstructed"] = [150.0, 250.0, float("nan")] + df["Volume_Source"] = ["reconstructed", "reconstructed", "raw"] + return df + + +def test_volume_defaults_to_front_month(tmp_store): + """The pre-existing shape. A caller that never heard of the parameter keeps + getting Norgate's continuous front-month volume.""" + store.write_bars("ES", _recon_frame(), domain=FUT, source="norgate", tier="backadj") + assert list(bars.get_bars("ES", "backadj")["Volume"]) == [100.0, 200.0, 300.0] + + +def test_reconstructed_volume_is_served_with_a_per_row_fallback(tmp_store): + """The producer writes the columns; this is the switch that serves them. + npf's ml/labels.py passes `volume=` through, so without it a repointed call + raises TypeError rather than returning the wrong number — but it still does + not work.""" + store.write_bars("ES", _recon_frame(), domain=FUT, source="norgate", tier="backadj") + out = bars.get_bars("ES", "backadj", volume="reconstructed") + + # Row 3 could not be reconstructed, so it falls back to front-month... + assert list(out["Volume"]) == [150.0, 250.0, 300.0] + # ...and says so, which is what lets a consumer exclude it. + assert list(out["Volume_Source"]) == ["reconstructed", "reconstructed", "raw"] + + +def test_a_store_predating_reconstruction_degrades_to_raw_and_says_so(tmp_store): + store.write_bars("ES", frame([1.0, 2.0]), domain=FUT, source="norgate", tier="backadj") + out = bars.get_bars("ES", "backadj", volume="reconstructed") + assert list(out["Volume_Source"]) == ["raw", "raw"] + assert list(out["Volume"]) == [100.0, 100.0] + + +def test_reconstructed_volume_is_refused_on_equities(tmp_store): + """It sums two expiries. An equity has none, and silently returning front-month + would answer a question that was not asked.""" + store.write_bars("SPY", frame([1.0]), domain="equities", source="yfinance") + with pytest.raises(ValueError, match="futures concept"): + bars.get_bars("SPY", "split", volume="reconstructed") + + +def test_an_unknown_volume_series_is_refused(tmp_store): + store.write_bars("ES", frame([1.0]), domain=FUT, source="norgate", tier="backadj") + with pytest.raises(ValueError, match="front"): + bars.get_bars("ES", "backadj", volume="whole_market") + + +def test_reconstructed_is_a_subset_not_whole_market(tmp_store): + """The naming trap, pinned. `reconstructed` sums the two biggest expiries and + is therefore SMALLER than front-month-plus-everything-else; `front` is what a + whole-market denominator wants. crowdmon's volume.py refuses anything else for + exactly this reason, and the test exists so the docstring cannot drift from it.""" + df = frame([1.0]) + df["Volume"] = [1000.0] + df["Volume_Reconstructed"] = [520.0] # ~natural gas's measured 0.52 ratio + df["Volume_Source"] = ["reconstructed"] + store.write_bars("NG", df, domain=FUT, source="norgate", tier="backadj") + + front = bars.get_bars("NG", "backadj")["Volume"].iloc[0] + recon = bars.get_bars("NG", "backadj", volume="reconstructed")["Volume"].iloc[0] + assert recon < front diff --git a/tests/test_verify_against_cotdata.py b/tests/test_verify_against_cotdata.py index 049bd09..9a4379d 100644 --- a/tests/test_verify_against_cotdata.py +++ b/tests/test_verify_against_cotdata.py @@ -248,3 +248,37 @@ def test_a_nonexistent_store_exits_two_rather_than_passing(verify, tmp_path): """A typo'd path must not read as 'nothing differed'.""" assert verify.main(["--cotdata-store", str(tmp_path / "nope"), "--marketdata-store", str(tmp_path)]) == 2 + + +# ── Coverage must be visible, or a PASS overstates itself ───────────────── +def test_a_column_missing_from_one_store_is_named_not_silently_skipped(verify): + """The flaw a real run exposed. Reconstruction columns were reported only when + they DIFFERED, so 'compared and identical' and 'never compared' printed the + same nothing — and a PASS could not be told apart from a PASS that skipped + half the frame.""" + cot = frame([1.0, 2.0]) + cot["Volume_Reconstructed"] = [10.0, 20.0] + mkt = frame([1.0, 2.0]) # no reconstruction columns at all + + rep = verify.compare_tier(cot, mkt) + assert rep["ok"] # passthrough still agrees + assert any("Volume_Reconstructed" in s and "only in cotdata" in s + for s in rep["not_compared"]) + + +def test_columns_in_neither_store_are_also_named(verify): + rep = verify.compare_tier(frame([1.0]), frame([1.0])) + assert rep["ok"] + assert any("in neither store" in s for s in rep["not_compared"]) + + +def test_nothing_is_flagged_uncompared_when_both_stores_carry_it(verify): + cot = frame([1.0, 2.0]) + for col, val in (("Volume_Reconstructed", 10.0), ("FirstVolume", 6.0), + ("SecondVolume", 4.0), ("FirstContract", "ES-2020H"), + ("SecondContract", "ES-2020M"), ("Volume_Source", "reconstructed")): + cot[col] = val + rep = verify.compare_tier(cot, cot.copy()) + assert rep["ok"] + assert rep["not_compared"] == [] + assert set(rep["reconstruction"]) == set(verify.RECONSTRUCTION) From 2295dc0532f3bcdd2722c998b8194cee6e283b9e Mon Sep 17 00:00:00 2001 From: "Matt S." Date: Sun, 9 Aug 2026 02:03:37 +0000 Subject: [PATCH 5/6] Record the verification result: identical across 49,892 rows per tier MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Run on the Windows producer against a cotdata store built by the original producer. Every compared series identical, exit 0, over five symbols spanning an index, an energy, a metal and the two markets whose backadj history goes non-positive. Exact equality rather than a tolerance, since both producers drive the same Norgate install through two code paths. This is what ADR-0007 §7.5 needs before cotdata's price code is deleted, and it was obtainable only while both halves still exist. Two things recorded beyond the verdict. The reconstruction columns matched, which this repo's own docs said to expect drift on. Both producers reconstruct incrementally over their own store's history, so a fresh store recomputing 12,000 bars against one that accumulated them over months looked like a real source of difference. It is not, and the reason generalises: Norgate's historical individual-contract volumes are immutable and the algorithm is identical, so incremental converges on full. That makes --strict-volume usable rather than theoretical. And the two defects the real box found were both invisible to the offline suite — a dependency no extra declared, and a producer-side column family with no consumer-side switch to serve it. A test that cannot install the vendor cannot catch the first, and no offline test calls a parameter that does not exist. The harness caught the second only because it was changed to report the columns it had NOT compared rather than staying silent about them. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Jo4iovRfc2fzE9MwcLp7r2 --- docs/design.md | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/docs/design.md b/docs/design.md index 2204dff..43c78a4 100644 --- a/docs/design.md +++ b/docs/design.md @@ -171,6 +171,44 @@ use — they read a **synced** one. That is the answer, not a temporary state: failing, and `--domain futures` on such a box explains why instead of raising `ModuleNotFoundError`. +### Verified against cotdata, 2026-08-09 + +Run on the Windows producer with `scripts/verify_against_cotdata.py`, against a +cotdata store built by the original producer. **Every compared series identical, +exit 0.** + +| symbol | rows per tier | passthrough | reconstruction | +|---|---:|---|---| +| ES | 7,279 | identical | identical | +| CL | 10,887 | identical | identical | +| GC | 12,156 | identical | identical | +| ZS | 12,271 | identical | identical | +| DC | 7,299 | identical | identical | + +49,892 rows per tier, both tiers, plus contract specs for all five. Exact +equality, not a tolerance: the two producers drive the same Norgate install +through two code paths, so any difference would have been a port bug rather than +vendor disagreement. + +Two things worth recording beyond the verdict. + +**The reconstruction columns matched too, which was not expected.** Both +producers reconstruct volume incrementally over their own store's history, so a +fresh marketdata store recomputing 12,000 bars and a cotdata store that +accumulated them over months looked like a legitimate source of drift. They agree +exactly, and the reason holds generally: Norgate's historical individual-contract +volumes are immutable and the algorithm is the same, so the incremental path +converges on what a full recompute produces. `--strict-volume` is therefore +usable rather than theoretical. + +**The first two runs found defects offline testing could not.** `--domain +futures` stopped at the import guard because the `norgate` extra was never +declared, and the reconstruction columns turned out to have no consumer-side +`volume=` switch — the producer wrote them and nothing served them. Neither is +visible to a test suite that cannot install the vendor or call the missing +parameter. The comparison harness is what caught the second, by reporting which +columns it had NOT compared instead of staying silent about them. + ### What is NOT ported `MME` and `MFS` (MSCI EM and EAFE). Norgate carries no continuous series for From 0cadae042666af09ae57ae7e789438799b4c6250 Mon Sep 17 00:00:00 2001 From: "Matt S." Date: Sun, 9 Aug 2026 02:11:29 +0000 Subject: [PATCH 6/6] Check the norgate extra without tomllib, which 3.10 does not have MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI failed on test (3.10) alone. tomllib is stdlib only from 3.11, and this package declares >=3.10, so the test I added to stop the packaging gap recurring could not run on the floor version — the one most likely to be a stale producer environment, and therefore the one the check is worth most on. Reading the file as text asserts the same fact on every supported version. It is also honest about what the test is: a check that a line is declared in packaging metadata, not a check of TOML semantics. Verified on 3.10.20 as well as 3.11 this time, rather than on the interpreter that happened to be in the venv. That is the same class of mistake as the two the Windows box found — a green suite proving something narrower than it appeared to. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Jo4iovRfc2fzE9MwcLp7r2 --- tests/test_futures.py | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/tests/test_futures.py b/tests/test_futures.py index 1c04925..52ebaeb 100644 --- a/tests/test_futures.py +++ b/tests/test_futures.py @@ -421,16 +421,19 @@ def test_the_norgate_extra_is_declared(): This shipped missing: the futures provider landed with no `norgate` extra, so `--domain futures` on the Windows producer stopped at the import guard with the package never having been pulled. The guard did its job; the packaging had not. + + Read as text rather than parsed: `tomllib` is stdlib only from 3.11 and this + package supports 3.10, so parsing would skip the check on the floor version — + the one most likely to be a stale producer environment. """ + import re from pathlib import Path - import tomllib - - pyproject = Path(__file__).resolve().parents[1] / "pyproject.toml" - extras = tomllib.loads(pyproject.read_text())["project"]["optional-dependencies"] - assert "norgate" in extras, ( + pyproject = (Path(__file__).resolve().parents[1] / "pyproject.toml").read_text() + declared = re.search(r"^norgate\s*=\s*\[([^\]]*)\]", pyproject, re.M) + assert declared, ( "providers/norgate.py imports norgatedata, but no extra installs it") - assert any("norgatedata" in dep for dep in extras["norgate"]) + assert "norgatedata" in declared.group(1) def test_the_missing_package_message_names_the_extra_that_fixes_it():