diff --git a/README.md b/README.md index 14124e1..d418045 100644 --- a/README.md +++ b/README.md @@ -23,6 +23,17 @@ moves every bar here. uv venv --python 3.11 && uv pip install -e ".[yahoo,dev]" "setuptools<81" ``` +On the **Windows futures producer**, add the `norgate` extra — nothing else pulls +`norgatedata`, and without it `--domain futures` stops before it fetches: + +```bash +uv pip install -e ".[yahoo,norgate,dev]" "setuptools<81" +``` + +Installing it elsewhere does not help. It drives a locally installed Norgate Data +Updater rather than an API, and NDU is Windows-only, so every other machine reads +a synced store instead of producing one. + ## Use ```bash @@ -151,6 +162,44 @@ silently substituting a vendor is what ADR-0006 forbids. The store root may share a parent folder with cotdata's, but the two must not share a `manifest.json`. Both producers do a read-modify-write on it. +### On the Windows futures producer + +That box now runs **two producers**, so it needs **both** store variables set at +once, pointing at **different roots**. This is new with the futures domain: until +ADR-0007 moved bars here, `COTDATA_STORE` alone was the whole story. + +```cmd +setx COTDATA_STORE C:\Users\YourUsername\cotdata_store +setx MARKETDATA_STORE C:\Users\YourUsername\marketdata_store +``` + +`setx` persists; plain `set` lasts only for the current Command Prompt, which is +the usual reason a scheduled task cannot find a store an interactive shell could. +Open a NEW prompt afterwards — `setx` does not affect the one you typed it in — +and verify: + +```cmd +echo %COTDATA_STORE% +echo %MARKETDATA_STORE% +marketdata-update --check +``` + +`--check` reads the manifest and no network, so it is the cheap confirmation that +the variable points where you think. An unset variable is refused by name rather +than defaulted, because a silent default would write a second store somewhere +nobody looks. + +**Do not point them at one root.** Sharing a parent folder is fine and makes the +pair easy to sync; sharing a root is not, because both packages keep a +`manifest.json` at their root and each does a read-modify-write on it, so the two +producers would eventually drop each other's entries. + +Python, virtualenv and Task Scheduler setup are identical to cotdata's and are +not duplicated here — see +[cotdata's Windows setup guide](https://github.com/mspinola/cotdata/blob/main/docs/WINDOWS_SETUP.md). +The only marketdata-specific pieces are the `norgate` extra in **Install** above +and the two variables here. + ## Tests ```bash diff --git a/docs/design.md b/docs/design.md index 2204dff..43c78a4 100644 --- a/docs/design.md +++ b/docs/design.md @@ -171,6 +171,44 @@ use — they read a **synced** one. That is the answer, not a temporary state: failing, and `--domain futures` on such a box explains why instead of raising `ModuleNotFoundError`. +### Verified against cotdata, 2026-08-09 + +Run on the Windows producer with `scripts/verify_against_cotdata.py`, against a +cotdata store built by the original producer. **Every compared series identical, +exit 0.** + +| symbol | rows per tier | passthrough | reconstruction | +|---|---:|---|---| +| ES | 7,279 | identical | identical | +| CL | 10,887 | identical | identical | +| GC | 12,156 | identical | identical | +| ZS | 12,271 | identical | identical | +| DC | 7,299 | identical | identical | + +49,892 rows per tier, both tiers, plus contract specs for all five. Exact +equality, not a tolerance: the two producers drive the same Norgate install +through two code paths, so any difference would have been a port bug rather than +vendor disagreement. + +Two things worth recording beyond the verdict. + +**The reconstruction columns matched too, which was not expected.** Both +producers reconstruct volume incrementally over their own store's history, so a +fresh marketdata store recomputing 12,000 bars and a cotdata store that +accumulated them over months looked like a legitimate source of drift. They agree +exactly, and the reason holds generally: Norgate's historical individual-contract +volumes are immutable and the algorithm is the same, so the incremental path +converges on what a full recompute produces. `--strict-volume` is therefore +usable rather than theoretical. + +**The first two runs found defects offline testing could not.** `--domain +futures` stopped at the import guard because the `norgate` extra was never +declared, and the reconstruction columns turned out to have no consumer-side +`volume=` switch — the producer wrote them and nothing served them. Neither is +visible to a test suite that cannot install the vendor or call the missing +parameter. The comparison harness is what caught the second, by reporting which +columns it had NOT compared instead of staying silent about them. + ### What is NOT ported `MME` and `MFS` (MSCI EM and EAFE). Norgate carries no continuous series for diff --git a/pyproject.toml b/pyproject.toml index a0f515a..b3c9a6b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -32,6 +32,13 @@ Issues = "https://github.com/mspinola/marketdata/issues" # history(auto_adjust=...) has changed across yfinance versions — see # docs/design.md. Verified against 1.5.2. yahoo = ["yfinance>=1.5.2"] +# The futures producer, WINDOWS ONLY. The package installs anywhere, but it drives +# a locally installed Norgate Data Updater rather than an API, and NDU exists for +# Windows alone — so on a Mac or Linux box it imports fine and then has nothing to +# talk to. That is why this is an extra rather than a dependency, and why the +# provider guards the import and the service separately: a missing package and a +# stopped NDU are different problems with different fixes. +norgate = ["norgatedata"] dev = ["pytest>=7", "ruff==0.15.22"] [project.scripts] diff --git a/scripts/verify_against_cotdata.py b/scripts/verify_against_cotdata.py new file mode 100644 index 0000000..b7ca2f1 --- /dev/null +++ b/scripts/verify_against_cotdata.py @@ -0,0 +1,357 @@ +#!/usr/bin/env python +"""ADR-0007 step 2: prove the futures port preserved the numbers. + +Run this on the Windows box after a `marketdata-update --bars --domain futures`, +while `cotdata`'s price code still exists. Both producers read the same Norgate +install and write separate stores, so the two can be compared directly — until +ADR-0007 §7.5 deletes cotdata's half, at which point this check is gone forever. + +WHY IT MATTERS MORE THAN IT LOOKS. `crowdmon` was the consumer whose tier +requirements were strictest, and the original work order put it first precisely +so it would fail loudly if the provider were wrong. It was deprecated on +2026-08-07 with no consumers left, so that check is gone, and nothing exercises +`propadj` until the deferred `npf` pass. This script is what replaces it. + +WHAT AGREEMENT TO EXPECT — and it is much stricter than the databento harness +next door in cotdata. That one compares two INDEPENDENT vendors, whose roll +calendars and back-adjust anchors legitimately differ, so only the SHAPE can +agree. This compares the same vendor, the same symbol and the same adjustment +through two code paths. The vendor passthrough columns must be **exactly** equal. +A difference is a port bug, not a tolerance question. + +The one column family that may legitimately differ is the volume reconstruction +(`Volume_Reconstructed`, `FirstVolume`, `SecondVolume`, the contract names). +Both producers compute it identically, but incrementally: each only recomputes a +trailing window over what its own store already holds. A fresh marketdata store +recomputes the whole history under today's logic, while cotdata's has accumulated +over months of runs. So those columns are compared and REPORTED, not failed on — +see --strict-volume, and re-run cotdata with `--full` if you want them to match. + +Usage: + python scripts/verify_against_cotdata.py \\ + --cotdata-store "%COTDATA_STORE%" \\ + --marketdata-store "%MARKETDATA_STORE%" \\ + --symbols ES CL GC ZS DC + +Exit code is 0 only if every check passed, so it can gate the promotion. +Reads parquet directly from both stores: no network, and it does not need +`marketdata` or `cotdata` importable except for the optional --check-propadj. +""" +from __future__ import annotations + +import argparse +import sys +from pathlib import Path + +import pandas as pd + +# The columns Norgate hands over and both producers store verbatim. These must be +# identical: both paths call the same `norgatedata.price_timeseries` and rename +# with the same map, so any difference here is the port having changed the data. +PASSTHROUGH = ("Open", "High", "Low", "Close", "Volume", "Open Interest", + "Delivery Month") + +# Computed identically by both, but over each store's own incremental window. +# Compared and reported rather than failed on. See the module docstring. +RECONSTRUCTION = ("Volume_Reconstructed", "FirstVolume", "SecondVolume", + "FirstContract", "SecondContract", "Volume_Source") + +STORED_TIERS = ("backadj", "unadj") + +# Symbols cotdata prices off ETF proxies through yfinance because Norgate carries +# no continuous series. Absent from marketdata's futures registry on purpose, so +# their absence is reported as expected rather than as a missing symbol. +EXPECTED_ABSENT = ("MME", "MFS") + + +def _norm(df: pd.DataFrame) -> pd.DataFrame: + df = df.copy() + df.index = pd.to_datetime(df.index).tz_localize(None).normalize() + df.index.name = "Date" + return df.sort_index() + + +def read_cotdata(store: Path, symbol: str, tier: str) -> pd.DataFrame: + """cotdata layout: prices/_.parquet""" + p = store / "prices" / f"{symbol}_{tier}.parquet" + return _norm(pd.read_parquet(p)) if p.exists() else pd.DataFrame() + + +def read_marketdata(store: Path, symbol: str, tier: str) -> pd.DataFrame: + """marketdata layout: bars/futures/norgate/_.parquet""" + p = store / "bars" / "futures" / "norgate" / f"{symbol}_{tier}.parquet" + return _norm(pd.read_parquet(p)) if p.exists() else pd.DataFrame() + + +def compare_column(a: pd.Series, b: pd.Series) -> dict: + """Exact-equality report for one column over a shared index. + + NaN == NaN counts as equal: a missing Open Interest that is missing in both + stores is agreement, and `!=` would call it a difference on every such row. + """ + both_null = a.isna() & b.isna() + if pd.api.types.is_numeric_dtype(a) and pd.api.types.is_numeric_dtype(b): + diff = (a - b).abs() + unequal = ~(both_null | (diff == 0)) + worst = float(diff[unequal].max()) if unequal.any() else 0.0 + else: + unequal = ~(both_null | (a.astype("object") == b.astype("object"))) + worst = float("nan") if unequal.any() else 0.0 + n = int(unequal.sum()) + return { + "n_differing": n, + "worst_abs_diff": worst, + "first_date": str(a.index[unequal][0].date()) if n else None, + } + + +def compare_tier(cot: pd.DataFrame, mkt: pd.DataFrame) -> dict: + """One stored tier, one symbol. Returns a report dict; `ok` is the verdict on + the passthrough columns only.""" + rep: dict = {"cot_rows": len(cot), "mkt_rows": len(mkt), + "passthrough": {}, "reconstruction": {}, "problems": [], + "not_compared": []} + if cot.empty or mkt.empty: + rep["problems"].append( + "absent from " + ("cotdata" if cot.empty else "marketdata")) + rep["ok"] = False + return rep + + rep["cot_span"] = f"{cot.index.min().date()}..{cot.index.max().date()}" + rep["mkt_span"] = f"{mkt.index.min().date()}..{mkt.index.max().date()}" + + common = cot.index.intersection(mkt.index) + rep["n_common"] = len(common) + # Dates one store has and the other does not. A handful at the tail is just + # one producer having run more recently; a gap in the middle is a real + # difference in what was captured. + rep["cot_only"] = len(cot.index.difference(mkt.index)) + rep["mkt_only"] = len(mkt.index.difference(cot.index)) + if len(common) == 0: + rep["problems"].append("no overlapping dates") + rep["ok"] = False + return rep + + c, m = cot.loc[common], mkt.loc[common] + for col in PASSTHROUGH: + if col not in c.columns and col not in m.columns: + continue + if col not in c.columns or col not in m.columns: + rep["problems"].append( + f"{col}: present in only one store " + f"({'cotdata' if col in c.columns else 'marketdata'})") + continue + r = compare_column(c[col], m[col]) + rep["passthrough"][col] = r + if r["n_differing"]: + rep["problems"].append( + f"{col}: {r['n_differing']} of {len(common)} rows differ " + f"(worst {r['worst_abs_diff']}, first {r['first_date']})") + + # Coverage is reported, not just differences. A column absent from one store + # is skipped, and if that skip were silent it would look exactly like a + # column that matched — so a PASS could not be read as "everything was + # compared". Naming the untested columns is the difference between evidence + # and a green light. + for col in RECONSTRUCTION: + in_c, in_m = col in c.columns, col in m.columns + if in_c and in_m: + rep["reconstruction"][col] = compare_column(c[col], m[col]) + elif in_c or in_m: + rep["not_compared"].append( + f"{col} (only in {'cotdata' if in_c else 'marketdata'})") + else: + rep["not_compared"].append(f"{col} (in neither store)") + + rep["ok"] = not rep["problems"] + return rep + + +def compare_specs(cot_store: Path, mkt_store: Path, symbols) -> dict: + """contract_specs is one table keyed by Symbol, so compare row by row.""" + rep: dict = {"problems": [], "compared": 0} + cp = cot_store / "metadata" / "contract_specs.parquet" + mp = mkt_store / "metadata" / "contract_specs.parquet" + if not cp.exists() or not mp.exists(): + rep["problems"].append( + f"contract_specs missing in {'cotdata' if not cp.exists() else 'marketdata'} " + f"(run marketdata-update --metadata)") + rep["ok"] = False + return rep + + c = pd.read_parquet(cp).set_index("Symbol") + m = pd.read_parquet(mp).set_index("Symbol") + for sym in symbols: + if sym in EXPECTED_ABSENT: + continue + if sym not in c.index: + continue # cotdata never had it; not a port question + if sym not in m.index: + rep["problems"].append(f"{sym}: no contract_specs row in marketdata") + continue + rep["compared"] += 1 + for col in ("Point Value", "Tick Size", "Tick Value", "Currency", + "Exchange", "Name"): + if col not in c.columns or col not in m.columns: + continue + cv, mv = c.loc[sym, col], m.loc[sym, col] + if pd.isna(cv) and pd.isna(mv): + continue + if cv != mv: + rep["problems"].append(f"{sym}.{col}: cotdata {cv!r}, marketdata {mv!r}") + rep["ok"] = not rep["problems"] + return rep + + +def check_propadj(symbol: str, cot_store: Path) -> dict: + """Run both ratio-adjust implementations over the SAME input frames. + + Store equality is checked separately, so feeding both implementations one pair + of frames isolates the ALGORITHM port from the data. Needs both packages + importable; skipped otherwise. + """ + rep: dict = {"skipped": None, "problems": []} + try: + from cotdata.prices import _ratio_adjust as cot_ratio + + from marketdata.adjust import ratio_adjust as mkt_ratio + except ImportError as e: + rep["skipped"] = f"{e} (needs both packages importable)" + return rep + + unadj = read_cotdata(cot_store, symbol, "unadj") + backadj = read_cotdata(cot_store, symbol, "backadj") + if unadj.empty or backadj.empty: + rep["skipped"] = "cotdata store lacks both tiers for this symbol" + return rep + + mine = mkt_ratio(unadj, backadj) + # cotdata's reads the store itself and keys off COTDATA_STORE, so point it at + # the same store rather than reimplementing its read. + theirs = cot_ratio(symbol) + if theirs.empty: + rep["skipped"] = "cotdata's _ratio_adjust returned empty (is COTDATA_STORE set?)" + return rep + + common = mine.index.intersection(theirs.index) + rep["n_common"] = len(common) + if not len(common): + rep["problems"].append("no overlapping dates between the two propadj series") + rep["ok"] = False + return rep + for col in ("Open", "High", "Low", "Close"): + if col in mine.columns and col in theirs.columns: + d = (mine.loc[common, col] - theirs.loc[common, col]).abs().max() + rep[col] = float(d) + # Same algorithm on the same inputs: floating-point noise only. + if pd.notna(d) and d > 1e-9: + rep["problems"].append(f"{col}: max abs diff {d:g} exceeds 1e-9") + rep["ok"] = not rep["problems"] + return rep + + +def main(argv=None) -> int: + p = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + p.add_argument("--cotdata-store", required=True) + p.add_argument("--marketdata-store", required=True) + p.add_argument("--symbols", nargs="+", default=["ES", "CL", "GC", "ZS", "DC"], + help="default set spans an index, an energy, a metal, and the two " + "markets whose backadj history goes non-positive") + p.add_argument("--strict-volume", action="store_true", + help="fail on reconstructed-volume differences too. Only meaningful " + "after a cotdata run with --full, since both producers " + "reconstruct incrementally over their own store's history") + p.add_argument("--check-propadj", action="store_true", + help="also run cotdata's and marketdata's ratio-adjust over the same " + "frames. Needs both packages importable") + args = p.parse_args(argv) + + cot_store, mkt_store = Path(args.cotdata_store), Path(args.marketdata_store) + for label, path in (("cotdata", cot_store), ("marketdata", mkt_store)): + if not path.exists(): + print(f"ERROR: {label} store does not exist: {path}") + return 2 + + failures, untested = [], [] + print(f"cotdata store {cot_store}") + print(f"marketdata store {mkt_store}\n") + + for sym in args.symbols: + if sym in EXPECTED_ABSENT: + print(f"{sym}: skipped — Norgate carries no continuous series, priced off " + f"an ETF proxy in cotdata and deliberately not ported") + continue + print(f"── {sym} " + "─" * 60) + + for tier in STORED_TIERS: + rep = compare_tier(read_cotdata(cot_store, sym, tier), + read_marketdata(mkt_store, sym, tier)) + status = "OK " if rep["ok"] else "FAIL" + head = (f" {status} {tier:8s} cot {rep['cot_rows']:6d} rows, " + f"mkt {rep['mkt_rows']:6d}") + if "n_common" in rep: + head += (f", {rep['n_common']} common" + + (f", {rep['cot_only']} cot-only" if rep["cot_only"] else "") + + (f", {rep['mkt_only']} mkt-only" if rep["mkt_only"] else "")) + print(head) + for prob in rep["problems"]: + print(f" ! {prob}") + if not rep["ok"]: + failures.append(f"{sym}/{tier}") + + matched = [c for c, r in rep["reconstruction"].items() + if not r["n_differing"]] + if matched: + print(f" ok reconstruction identical: {', '.join(matched)}") + for col, r in rep["reconstruction"].items(): + if r["n_differing"]: + note = ("FAIL" if args.strict_volume else "note") + print(f" {note} {col}: {r['n_differing']} rows differ " + f"(first {r['first_date']}) — expected unless cotdata was " + f"last run with --full") + if args.strict_volume: + failures.append(f"{sym}/{tier}/{col}") + if rep["not_compared"]: + print(f" ?? NOT COMPARED: {'; '.join(rep['not_compared'])}") + untested.append(f"{sym}/{tier}") + + if args.check_propadj: + r = check_propadj(sym, cot_store) + if r.get("skipped"): + print(f" SKIP propadj {r['skipped']}") + else: + print(f" {'OK ' if r['ok'] else 'FAIL'} propadj " + f"{r.get('n_common', 0)} common rows, " + f"max close diff {r.get('Close', float('nan')):g}") + for prob in r["problems"]: + print(f" ! {prob}") + if not r["ok"]: + failures.append(f"{sym}/propadj") + + print("\n── contract specs " + "─" * 51) + specs = compare_specs(cot_store, mkt_store, args.symbols) + print(f" {'OK ' if specs['ok'] else 'FAIL'} {specs['compared']} symbols compared") + for prob in specs["problems"]: + print(f" ! {prob}") + if not specs["ok"]: + failures.append("contract_specs") + + print() + if untested: + print(f"NOTE: {len(untested)} tier(s) had columns present in only one store, or " + f"in neither, and those columns were NOT compared — see '?? NOT COMPARED' " + f"above. The verdict below covers what was compared.") + if failures: + print(f"FAILED: {len(failures)} check(s) — {', '.join(failures)}") + print("\nThe two producers read the same Norgate install, so the passthrough " + "columns differing means the port changed the data. Do not promote.") + return 1 + print("PASSED: every compared series is identical between the two stores.") + print("\nThat is the evidence ADR-0007 §7.5 needs before cotdata's price code is " + "deleted, and it is only obtainable while both halves still exist.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/src/marketdata/bars.py b/src/marketdata/bars.py index bf6cf50..af49e04 100644 --- a/src/marketdata/bars.py +++ b/src/marketdata/bars.py @@ -32,6 +32,7 @@ def default_source_for(symbol: str) -> str: def get_bars(symbol: str, adjustment: Optional[str] = None, *, source: Optional[str] = None, domain: Optional[str] = None, start: Optional[str] = None, end: Optional[str] = None, + volume: str = "front", include_capital_gains: bool = False) -> pd.DataFrame: """Daily bars for `symbol`, adjusted to `adjustment`. @@ -56,6 +57,23 @@ def get_bars(symbol: str, adjustment: Optional[str] = None, *, where additive adjustment has driven the back-history through zero (ZS, DC). See `adjust.ratio_adjust`. + `volume` selects which series the `Volume` column carries (futures only): + 'front' : continuous front-month volume as Norgate reports it. + Default. + 'reconstructed' : FirstVolume + SecondVolume — the two highest-volume + expiries trading that day, with a per-row fall-back to + front-month where individual contracts are unavailable. + Adds a `Volume_Source` column ('reconstructed' / 'raw') + so a consumer can tell the fall-back rows apart. + + **Read those two again: the names are a trap, and it points the opposite way + to intuition.** `reconstructed` is NOT whole-market. It sums exactly two + expiries, which is 0.52 of total volume in natural gas and 0.54 in crude, so + it understates most in precisely the markets with the deepest curves. + `front` is the series to divide a whole-market quantity by. Measured in + crowdmon, whose `futures/volume.py` refuses anything but `front` for that + reason; carried here because the naming will mislead the next reader too. + `domain` is resolved from the registry and rarely passed. `source` pins the vendor. Omit it and the registry resolves one for this deployment. Pass it explicitly to compare vendors on the same symbol, which is the point of @@ -71,6 +89,13 @@ def get_bars(symbol: str, adjustment: Optional[str] = None, *, check_tier(adjustment, dom) src = source or default_source_for(symbol) + if volume not in ("front", "reconstructed"): + raise ValueError(f"volume must be 'front' or 'reconstructed', got {volume!r}") + if volume == "reconstructed" and dom != "futures": + raise ValueError( + f"volume='reconstructed' is a futures concept (it sums the two " + f"highest-volume expiries) and {symbol!r} is in domain {dom!r}.") + if dom == "futures": out = _futures_bars(symbol, src, adjustment) else: @@ -78,6 +103,8 @@ def get_bars(symbol: str, adjustment: Optional[str] = None, *, include_capital_gains=include_capital_gains) if out.empty: return out + if volume == "reconstructed": + out = _reconstructed_volume(out) if start is not None: out = out[out.index >= pd.Timestamp(start)] @@ -106,6 +133,29 @@ def _missing(symbol: str, dom: str, src: str, tier=None) -> None: f"MARKETDATA_PRICE_SOURCE. Vendors are never silently substituted.") +def _reconstructed_volume(df: pd.DataFrame) -> pd.DataFrame: + """Swap `Volume` for the reconstructed series, with a per-row fall-back. + + The producer already writes `Volume_Reconstructed == Volume` on rows it could + not reconstruct, so reading the column is fall-back-safe where it exists. The + guards are for a store written before reconstruction, or a stray NaN: both + degrade to front-month volume and SAY SO in `Volume_Source`, because a + consumer comparing volume across symbols has to be able to exclude the rows + that are not really reconstructed. + """ + out = df.copy() + if "Volume_Reconstructed" in out.columns: + rec = out["Volume_Reconstructed"] + out["Volume"] = rec.where(rec.notna(), out["Volume"]) + if "Volume_Source" not in out.columns: + out["Volume_Source"] = "reconstructed" + out.loc[rec.isna(), "Volume_Source"] = "raw" + else: + # Store predates reconstruction, so every row is front-month. + out["Volume_Source"] = "raw" + return out + + def _equity_bars(symbol: str, dom: str, src: str, tier: str, *, include_capital_gains: bool) -> pd.DataFrame: """One stored frame, every tier derived from it.""" diff --git a/src/marketdata/providers/norgate.py b/src/marketdata/providers/norgate.py index b77ec30..c5497f6 100644 --- a/src/marketdata/providers/norgate.py +++ b/src/marketdata/providers/norgate.py @@ -87,9 +87,13 @@ def _require_norgate_service() -> None: # the normal state of every other machine. raise RuntimeError( "norgatedata is not installed, so this machine cannot produce futures " - "bars. It drives a local Norgate Data Updater install, which is " - "Windows-only — a Mac or Linux box reads a SYNCED store instead of " - "producing one. Use --domain equities here.") from e + "bars.\n" + " On the Windows producer, this is just the missing extra:\n" + " uv pip install -e \".[norgate]\"\n" + " Anywhere else, installing it will not help: it drives a local " + "Norgate Data Updater install and NDU is Windows-only, so a Mac or " + "Linux box reads a SYNCED store rather than producing one. Use " + "--domain equities there.") from e try: reachable = bool(norgatedata.status()) except BaseException: # noqa: BLE001 — never let the probe itself take us down diff --git a/tests/test_futures.py b/tests/test_futures.py index 43f7b0b..52ebaeb 100644 --- a/tests/test_futures.py +++ b/tests/test_futures.py @@ -412,3 +412,117 @@ def test_provenance_reports_each_futures_tier_separately(tmp_store): assert provenance("ES").n_rows == 3 assert provenance("ES", tier="unadj").n_rows == 2 assert "backadj" in provenance("ES").describe() + + +# ── Packaging ───────────────────────────────────────────────────────────── +def test_the_norgate_extra_is_declared(): + """A provider whose vendor package nothing installs is a provider nobody can run. + + This shipped missing: the futures provider landed with no `norgate` extra, so + `--domain futures` on the Windows producer stopped at the import guard with the + package never having been pulled. The guard did its job; the packaging had not. + + Read as text rather than parsed: `tomllib` is stdlib only from 3.11 and this + package supports 3.10, so parsing would skip the check on the floor version — + the one most likely to be a stale producer environment. + """ + import re + from pathlib import Path + + pyproject = (Path(__file__).resolve().parents[1] / "pyproject.toml").read_text() + declared = re.search(r"^norgate\s*=\s*\[([^\]]*)\]", pyproject, re.M) + assert declared, ( + "providers/norgate.py imports norgatedata, but no extra installs it") + assert "norgatedata" in declared.group(1) + + +def test_the_missing_package_message_names_the_extra_that_fixes_it(): + """Two different problems with two different fixes: a missing package is an + install away on Windows, and unfixable anywhere else. The message has to + separate them or it sends the Windows producer to --domain equities.""" + import builtins + + real_import = builtins.__import__ + + def no_norgatedata(name, *a, **kw): + if name == "norgatedata": + raise ImportError("No module named 'norgatedata'") + return real_import(name, *a, **kw) + + builtins.__import__ = no_norgatedata + try: + with pytest.raises(RuntimeError) as e: + nprov._require_norgate_service() + finally: + builtins.__import__ = real_import + + msg = str(e.value) + assert "[norgate]" in msg # the fix, for the box that can be fixed + assert "Windows" in msg # and why it is not the fix anywhere else + + +# ── Reconstructed volume, consumer side ─────────────────────────────────── +def _recon_frame(): + df = frame([1.0, 2.0, 3.0]) + df["Volume"] = [100.0, 200.0, 300.0] + df["Volume_Reconstructed"] = [150.0, 250.0, float("nan")] + df["Volume_Source"] = ["reconstructed", "reconstructed", "raw"] + return df + + +def test_volume_defaults_to_front_month(tmp_store): + """The pre-existing shape. A caller that never heard of the parameter keeps + getting Norgate's continuous front-month volume.""" + store.write_bars("ES", _recon_frame(), domain=FUT, source="norgate", tier="backadj") + assert list(bars.get_bars("ES", "backadj")["Volume"]) == [100.0, 200.0, 300.0] + + +def test_reconstructed_volume_is_served_with_a_per_row_fallback(tmp_store): + """The producer writes the columns; this is the switch that serves them. + npf's ml/labels.py passes `volume=` through, so without it a repointed call + raises TypeError rather than returning the wrong number — but it still does + not work.""" + store.write_bars("ES", _recon_frame(), domain=FUT, source="norgate", tier="backadj") + out = bars.get_bars("ES", "backadj", volume="reconstructed") + + # Row 3 could not be reconstructed, so it falls back to front-month... + assert list(out["Volume"]) == [150.0, 250.0, 300.0] + # ...and says so, which is what lets a consumer exclude it. + assert list(out["Volume_Source"]) == ["reconstructed", "reconstructed", "raw"] + + +def test_a_store_predating_reconstruction_degrades_to_raw_and_says_so(tmp_store): + store.write_bars("ES", frame([1.0, 2.0]), domain=FUT, source="norgate", tier="backadj") + out = bars.get_bars("ES", "backadj", volume="reconstructed") + assert list(out["Volume_Source"]) == ["raw", "raw"] + assert list(out["Volume"]) == [100.0, 100.0] + + +def test_reconstructed_volume_is_refused_on_equities(tmp_store): + """It sums two expiries. An equity has none, and silently returning front-month + would answer a question that was not asked.""" + store.write_bars("SPY", frame([1.0]), domain="equities", source="yfinance") + with pytest.raises(ValueError, match="futures concept"): + bars.get_bars("SPY", "split", volume="reconstructed") + + +def test_an_unknown_volume_series_is_refused(tmp_store): + store.write_bars("ES", frame([1.0]), domain=FUT, source="norgate", tier="backadj") + with pytest.raises(ValueError, match="front"): + bars.get_bars("ES", "backadj", volume="whole_market") + + +def test_reconstructed_is_a_subset_not_whole_market(tmp_store): + """The naming trap, pinned. `reconstructed` sums the two biggest expiries and + is therefore SMALLER than front-month-plus-everything-else; `front` is what a + whole-market denominator wants. crowdmon's volume.py refuses anything else for + exactly this reason, and the test exists so the docstring cannot drift from it.""" + df = frame([1.0]) + df["Volume"] = [1000.0] + df["Volume_Reconstructed"] = [520.0] # ~natural gas's measured 0.52 ratio + df["Volume_Source"] = ["reconstructed"] + store.write_bars("NG", df, domain=FUT, source="norgate", tier="backadj") + + front = bars.get_bars("NG", "backadj")["Volume"].iloc[0] + recon = bars.get_bars("NG", "backadj", volume="reconstructed")["Volume"].iloc[0] + assert recon < front diff --git a/tests/test_verify_against_cotdata.py b/tests/test_verify_against_cotdata.py new file mode 100644 index 0000000..9a4379d --- /dev/null +++ b/tests/test_verify_against_cotdata.py @@ -0,0 +1,284 @@ +"""The port-verification harness, tested against synthetic frames. + +The script itself needs two real stores and a Norgate install, so it cannot run in +CI. Its comparison logic can, and must: a verifier that reports "identical" on +frames that differ is worse than no verifier, because it would clear the port for +deletion of the only thing it could ever be compared against. + +Same posture as cotdata's `tests/test_validate_databento.py` beside the harness it +covers. +""" +import importlib.util +import sys +from pathlib import Path + +import pandas as pd +import pytest + +_SCRIPT = Path(__file__).resolve().parents[1] / "scripts" / "verify_against_cotdata.py" + + +@pytest.fixture(scope="module") +def verify(): + spec = importlib.util.spec_from_file_location("verify_against_cotdata", _SCRIPT) + mod = importlib.util.module_from_spec(spec) + sys.modules[spec.name] = mod + spec.loader.exec_module(mod) + return mod + + +def frame(closes, *, volume=100.0, delivery=None, start="2020-01-01"): + idx = pd.date_range(start, periods=len(closes), freq="D") + df = pd.DataFrame({"Open": closes, "High": closes, "Low": closes, + "Close": closes, "Volume": volume, + "Open Interest": 1000.0}, index=idx) + if delivery is not None: + df["Delivery Month"] = delivery + df.index.name = "Date" + return df + + +# ── The verdict has to be right in both directions ──────────────────────── +def test_identical_frames_pass(verify): + df = frame([1.0, 2.0, 3.0], delivery=["A", "A", "B"]) + rep = verify.compare_tier(df, df.copy()) + assert rep["ok"] + assert rep["problems"] == [] + assert rep["n_common"] == 3 + + +def test_a_single_changed_price_fails_and_is_located(verify): + """The check that earns the harness its keep. One wrong bar in ten thousand is + exactly what a silent port bug looks like.""" + cot = frame([1.0, 2.0, 3.0]) + mkt = cot.copy() + mkt.iloc[1, mkt.columns.get_loc("Close")] = 2.0001 + + rep = verify.compare_tier(cot, mkt) + assert not rep["ok"] + assert rep["passthrough"]["Close"]["n_differing"] == 1 + assert rep["passthrough"]["Close"]["first_date"] == "2020-01-02" + assert any("Close" in p for p in rep["problems"]) + + +def test_tolerance_is_exact_not_approximate(verify): + """Two INDEPENDENT vendors would need a tolerance. These are two code paths + over one vendor, so a float away is still a difference.""" + cot = frame([100.0]) + mkt = frame([100.0 + 1e-12]) + assert not verify.compare_tier(cot, mkt)["ok"] + + +def test_a_changed_delivery_month_fails(verify): + """Non-numeric, and it drives roll detection — so propadj is wrong downstream + of a difference here, silently.""" + cot = frame([1.0, 2.0], delivery=["2020H", "2020H"]) + mkt = frame([1.0, 2.0], delivery=["2020H", "2020M"]) + rep = verify.compare_tier(cot, mkt) + assert not rep["ok"] + assert rep["passthrough"]["Delivery Month"]["n_differing"] == 1 + + +def test_matching_nans_are_agreement_not_difference(verify): + """An Open Interest Norgate never supplied is missing in both stores. Comparing + with `!=` would call every such row a difference and drown the real ones.""" + cot = frame([1.0, 2.0]) + cot["Open Interest"] = [float("nan"), 5.0] + rep = verify.compare_tier(cot, cot.copy()) + assert rep["ok"] + assert rep["passthrough"]["Open Interest"]["n_differing"] == 0 + + +def test_a_nan_on_one_side_only_is_a_difference(verify): + cot = frame([1.0, 2.0]) + mkt = cot.copy() + mkt["Open Interest"] = [float("nan"), 1000.0] + assert not verify.compare_tier(cot, mkt)["ok"] + + +# ── Absence and partial overlap ─────────────────────────────────────────── +def test_missing_from_marketdata_fails_and_says_which_side(verify): + rep = verify.compare_tier(frame([1.0]), pd.DataFrame()) + assert not rep["ok"] + assert "marketdata" in rep["problems"][0] + + +def test_missing_from_cotdata_fails_and_says_which_side(verify): + rep = verify.compare_tier(pd.DataFrame(), frame([1.0])) + assert not rep["ok"] + assert "cotdata" in rep["problems"][0] + + +def test_differing_tails_are_counted_not_failed(verify): + """One producer having run more recently is normal. Only the OVERLAP is + compared, and the extra days are reported so a gap in the middle is still + visible.""" + cot = frame([1.0, 2.0, 3.0, 4.0]) + mkt = frame([1.0, 2.0, 3.0]) + rep = verify.compare_tier(cot, mkt) + assert rep["ok"] + assert rep["n_common"] == 3 + assert rep["cot_only"] == 1 + assert rep["mkt_only"] == 0 + + +def test_no_overlap_at_all_fails(verify): + cot = frame([1.0, 2.0], start="2020-01-01") + mkt = frame([1.0, 2.0], start="2021-01-01") + rep = verify.compare_tier(cot, mkt) + assert not rep["ok"] + assert "no overlapping dates" in rep["problems"][0] + + +# ── Reconstruction columns are reported, never failed ───────────────────── +def test_reconstructed_volume_differences_do_not_fail_the_tier(verify): + """Both producers reconstruct incrementally over their own store's history, so + a fresh marketdata store and a months-old cotdata one legitimately differ here. + Reported for the operator; --strict-volume is the opt-in.""" + cot = frame([1.0, 2.0]) + cot["Volume_Reconstructed"] = [10.0, 20.0] + mkt = frame([1.0, 2.0]) + mkt["Volume_Reconstructed"] = [10.0, 999.0] + + rep = verify.compare_tier(cot, mkt) + assert rep["ok"], "a reconstruction difference must not fail the passthrough verdict" + assert rep["reconstruction"]["Volume_Reconstructed"]["n_differing"] == 1 + + +def test_a_price_difference_still_fails_alongside_a_reconstruction_difference(verify): + """The reconstruction leniency must not swallow a real one.""" + cot = frame([1.0, 2.0]) + cot["Volume_Reconstructed"] = [10.0, 20.0] + mkt = frame([1.0, 2.5]) + mkt["Volume_Reconstructed"] = [10.0, 999.0] + assert not verify.compare_tier(cot, mkt)["ok"] + + +# ── Store readers match each layout ─────────────────────────────────────── +def test_readers_use_each_stores_own_layout(verify, tmp_path): + """The two stores name files differently, and reading the wrong path would + report a symbol absent rather than compare it.""" + cot_store, mkt_store = tmp_path / "cot", tmp_path / "mkt" + (cot_store / "prices").mkdir(parents=True) + (mkt_store / "bars" / "futures" / "norgate").mkdir(parents=True) + + frame([1.0, 2.0]).to_parquet(cot_store / "prices" / "ES_backadj.parquet") + frame([1.0, 2.0]).to_parquet( + mkt_store / "bars" / "futures" / "norgate" / "ES_backadj.parquet") + + assert len(verify.read_cotdata(cot_store, "ES", "backadj")) == 2 + assert len(verify.read_marketdata(mkt_store, "ES", "backadj")) == 2 + assert verify.read_cotdata(cot_store, "ES", "unadj").empty + assert verify.read_marketdata(mkt_store, "GC", "backadj").empty + + +# ── Contract specs ──────────────────────────────────────────────────────── +def _write_specs(root, rows): + (root / "metadata").mkdir(parents=True, exist_ok=True) + pd.DataFrame(rows).to_parquet(root / "metadata" / "contract_specs.parquet") + + +def test_matching_specs_pass_and_a_changed_point_value_fails(verify, tmp_path): + """Point Value is the multiplier every notional and risk-unit figure scales by, + so a silent change here rescales an entire book.""" + cot, mkt = tmp_path / "cot", tmp_path / "mkt" + rows = [{"Symbol": "ES", "Point Value": 50.0, "Tick Size": 0.25, + "Currency": "USD"}] + _write_specs(cot, rows) + _write_specs(mkt, rows) + assert verify.compare_specs(cot, mkt, ["ES"])["ok"] + + _write_specs(mkt, [{"Symbol": "ES", "Point Value": 5.0, "Tick Size": 0.25, + "Currency": "USD"}]) + rep = verify.compare_specs(cot, mkt, ["ES"]) + assert not rep["ok"] + assert "Point Value" in rep["problems"][0] + + +def test_a_symbol_missing_from_marketdata_specs_fails(verify, tmp_path): + cot, mkt = tmp_path / "cot", tmp_path / "mkt" + _write_specs(cot, [{"Symbol": "ES", "Point Value": 50.0}]) + _write_specs(mkt, [{"Symbol": "GC", "Point Value": 100.0}]) + rep = verify.compare_specs(cot, mkt, ["ES"]) + assert not rep["ok"] + assert "no contract_specs row in marketdata" in rep["problems"][0] + + +def test_expected_absent_symbols_are_not_reported_as_missing(verify, tmp_path): + """MME/MFS have no Norgate series and are deliberately unported. Failing on + them would train the operator to ignore a red run.""" + cot, mkt = tmp_path / "cot", tmp_path / "mkt" + _write_specs(cot, [{"Symbol": "MME", "Point Value": 50.0}]) + _write_specs(mkt, [{"Symbol": "ES", "Point Value": 50.0}]) + assert verify.compare_specs(cot, mkt, ["MME"])["ok"] + assert "MME" in verify.EXPECTED_ABSENT + + +def test_missing_specs_table_is_a_failure_naming_the_fix(verify, tmp_path): + cot, mkt = tmp_path / "cot", tmp_path / "mkt" + _write_specs(cot, [{"Symbol": "ES", "Point Value": 50.0}]) + mkt.mkdir(parents=True) + rep = verify.compare_specs(cot, mkt, ["ES"]) + assert not rep["ok"] + assert "--metadata" in rep["problems"][0] + + +# ── Exit code is the gate ───────────────────────────────────────────────── +def test_exit_code_is_zero_only_when_everything_matches(verify, tmp_path): + cot, mkt = tmp_path / "cot", tmp_path / "mkt" + (cot / "prices").mkdir(parents=True) + bars = mkt / "bars" / "futures" / "norgate" + bars.mkdir(parents=True) + for tier in ("backadj", "unadj"): + frame([1.0, 2.0]).to_parquet(cot / "prices" / f"ES_{tier}.parquet") + frame([1.0, 2.0]).to_parquet(bars / f"ES_{tier}.parquet") + specs = [{"Symbol": "ES", "Point Value": 50.0}] + _write_specs(cot, specs) + _write_specs(mkt, specs) + + argv = ["--cotdata-store", str(cot), "--marketdata-store", str(mkt), + "--symbols", "ES"] + assert verify.main(argv) == 0 + + frame([1.0, 99.0]).to_parquet(bars / "ES_unadj.parquet") + assert verify.main(argv) == 1 + + +def test_a_nonexistent_store_exits_two_rather_than_passing(verify, tmp_path): + """A typo'd path must not read as 'nothing differed'.""" + assert verify.main(["--cotdata-store", str(tmp_path / "nope"), + "--marketdata-store", str(tmp_path)]) == 2 + + +# ── Coverage must be visible, or a PASS overstates itself ───────────────── +def test_a_column_missing_from_one_store_is_named_not_silently_skipped(verify): + """The flaw a real run exposed. Reconstruction columns were reported only when + they DIFFERED, so 'compared and identical' and 'never compared' printed the + same nothing — and a PASS could not be told apart from a PASS that skipped + half the frame.""" + cot = frame([1.0, 2.0]) + cot["Volume_Reconstructed"] = [10.0, 20.0] + mkt = frame([1.0, 2.0]) # no reconstruction columns at all + + rep = verify.compare_tier(cot, mkt) + assert rep["ok"] # passthrough still agrees + assert any("Volume_Reconstructed" in s and "only in cotdata" in s + for s in rep["not_compared"]) + + +def test_columns_in_neither_store_are_also_named(verify): + rep = verify.compare_tier(frame([1.0]), frame([1.0])) + assert rep["ok"] + assert any("in neither store" in s for s in rep["not_compared"]) + + +def test_nothing_is_flagged_uncompared_when_both_stores_carry_it(verify): + cot = frame([1.0, 2.0]) + for col, val in (("Volume_Reconstructed", 10.0), ("FirstVolume", 6.0), + ("SecondVolume", 4.0), ("FirstContract", "ES-2020H"), + ("SecondContract", "ES-2020M"), ("Volume_Source", "reconstructed")): + cot[col] = val + rep = verify.compare_tier(cot, cot.copy()) + assert rep["ok"] + assert rep["not_compared"] == [] + assert set(rep["reconstruction"]) == set(verify.RECONSTRUCTION)