From 2abc5fae2ee2355d17fbfd414f0ccaba3c30f738 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 9 Aug 2026 13:37:49 +0000 Subject: [PATCH 1/2] =?UTF-8?q?ADR-0007=20=C2=A77.5:=20delete=20the=20pric?= =?UTF-8?q?e=20surface,=20keep=20databento?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit cotdata is CFTC positioning now. Gone: prices.py (with the derived propadj tier), providers/norgate.py, providers/yfinance.py, the get_prices/roll_dates exports, store.{write,upsert,read}_metadata, the --prices/--metadata/ --prices-yahoo/--require-final/--final-cutoff/--full flags, the norgate and yahoo extras, and the `packaging` dependency that existed only for norgatedata. Bars come from marketdata.get_bars against $MARKETDATA_STORE. Deleted rather than deprecated. A shim left importable would read a store the nightly job no longer fills, and stale data is much harder to notice than an AttributeError -- the failure is a number that looks right and is months old. A test asserts the names stay gone. DATABENTO STAYS, and with it store.write_prices/read_prices, config.prices_dir and the `prices` manifest half it writes through. It has no marketdata equivalent, and deleting it would destroy a validated provider-different alternative (ADR-0006) plus the only intraday-capable source in the fleet. So the CONSUMER bar API left and the store-level pair the retained producer needs did not -- read its output with store.read_prices. cotdata-prices survives, scoped to --ingest-databento/--build-databento. Worth stating plainly: "COT only" is now true of the public API and not yet of the store. The `metadata` manifest domain stays DECLARED with no writer, so stores written before this still migrate and reconcile their entries instead of stranding them in the legacy aggregate. Three things the deletion exposed rather than caused, each fixed first: * finals_ready() had no caller. It was ported in §7.1 but never wired to a CLI flag, so --require-final here was the only way to reach it -- deleting it would have left the Windows nightly job ungated, and a fetch before Norgate settles writes a provisional bar over a real one silently. Fixed as marketdata #13; this deletion was not safe without it. * Six behaviours had tests only here: volume reconstruction, the volume-rank pick, the incremental window, full=True, the NDU-down abort, the all-null spec-row skip. §7.1 ported the provider and not its tests. A file-count check would have missed it -- the test FILES existed on both sides, with zero name overlap and a real gap underneath. Ported in the same PR. * Two harnesses read the Norgate store by PATH, so no call-site grep finds them. validate_databento_vs_norgate.py and investigate_databento_roll_rule .py now read bars/futures/norgate/ first and fall back to the old layout, so the ADR-0006 parity gate still runs across the split. test_every_action_flag_is_assigned_to_a_half was reading a list copied beside the parser, which stayed green through this change while naming three flags that no longer exist -- so it could not have caught a fourth being added either. It now derives the flags from the parser, which needed update.main's argparse block extracted into _parser(). Known breakage, deliberate: crowdmon (frozen, archived) and npf's docs/crowdmon/reproduce_forced_flow_mechanism.py still call get_prices. Both are point-in-time records under their repos' doc lifecycle; §8.2 of the work order already voided the crowdmon repoint on the same grounds. Verified: ruff clean; 254 pass here. npf's full suite run against this branch and against main gives byte-identical failure sets (38/38, all pre-existing sibling-checkout gaps in this sandbox), and its contract-specs loader now resolves unambiguously to marketdata instead of warning that both packages could answer. --- CHANGELOG.md | 48 ++ CONTRIBUTING.md | 11 +- README.md | 200 ++++--- docs/LINUX_SCHEDULING.md | 12 +- docs/SYNCING.md | 32 +- docs/WINDOWS_SCHEDULING.md | 50 +- docs/WINDOWS_SETUP.md | 59 +- docs/databento_norgate_parity.md | 12 +- docs/design/finals_ready_data_driven.md | 11 + docs/design/reading-the-store.md | 23 +- docs/examples/linux/run-prices.sh | 6 +- docs/examples/mac/pull-store.sh | 2 +- docs/examples/windows/push-to-server.cmd | 2 +- docs/examples/windows/run-prices.cmd | 22 +- ...6-08-04-adr7-step2-price-producer-split.md | 60 +- pyproject.toml | 10 +- requirements.txt | 8 +- scripts/investigate_databento_roll_rule.py | 15 +- scripts/probe_norgate_finals.py | 68 --- scripts/reconcile_volume.py | 117 ---- scripts/validate_databento_vs_norgate.py | 41 +- src/cotdata/__init__.py | 13 +- src/cotdata/config.py | 11 +- src/cotdata/prices.py | 186 ------ src/cotdata/providers/norgate.py | 546 ------------------ src/cotdata/providers/yfinance.py | 70 --- src/cotdata/registry.py | 8 + src/cotdata/store.py | 49 +- src/cotdata/update.py | 117 ++-- tests/test_adjustment.py | 111 ---- tests/test_cli_exit.py | 44 +- tests/test_databento_build.py | 35 +- tests/test_manifest_seam.py | 21 +- tests/test_norgate_provider.py | 520 ----------------- tests/test_propadj.py | 142 ----- tests/test_store.py | 127 +--- tests/test_validate_databento.py | 14 +- tests/test_yfinance_provider.py | 61 -- 38 files changed, 610 insertions(+), 2274 deletions(-) delete mode 100644 scripts/probe_norgate_finals.py delete mode 100644 scripts/reconcile_volume.py delete mode 100644 src/cotdata/prices.py delete mode 100644 src/cotdata/providers/norgate.py delete mode 100644 src/cotdata/providers/yfinance.py delete mode 100644 tests/test_adjustment.py delete mode 100644 tests/test_norgate_provider.py delete mode 100644 tests/test_propadj.py delete mode 100644 tests/test_yfinance_provider.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 7f5e2f9..6a1f8b8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,54 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [Unreleased] +### Removed — BREAKING + +- **Price bars, and the Norgate and Yahoo producers, are gone from this package** + (crucible-stack ADR-0007 step 2 §7.5). cotdata is now CFTC positioning only. Every bar, + tier and contract spec lives in + [`crucible-marketdata`](https://pypi.org/project/crucible-marketdata/), which keeps its + own store (`MARKETDATA_STORE`), its own producer and its own schedule. + + | gone from cotdata | use instead | + |---|---| + | `cotdata.get_prices(sym, adjustment, start=, volume=)` | `marketdata.get_bars(sym, tier, start=, volume=)` (`start`/`volume` are keyword-only there) | + | `cotdata.roll_dates(sym)` | no replacement — see below | + | `cotdata.store.read_metadata()` | `marketdata.read_metadata()` | + | `cotdata-update --prices` / `--metadata` / `--require-final` | `marketdata-update --bars --domain futures --require-final`, `--metadata` | + | `cotdata-update --prices-yahoo` | `marketdata-update --bars --domain equities` | + | `cotdata[norgate]`, `cotdata[yahoo]` extras | `crucible-marketdata[norgate]`, `crucible-marketdata[yahoo]` | + | `COTDATA_PRICE_SOURCE` | marketdata's registry resolves the vendor per symbol | + + Also removed: `cotdata/prices.py` (including the derived `propadj` tier), + `providers/norgate.py`, `providers/yfinance.py`, `store.{write,upsert,read}_metadata`, + and the `packaging` runtime dependency that existed only for `norgatedata`. + + **Deleted rather than deprecated.** A shim left importable would read a store the nightly + job no longer fills, and stale data is much harder to notice than an `AttributeError` — + the failure is a number that looks right and is months old. + + `roll_dates` is dropped outright rather than ported: a sweep of the four consumer repos + found no caller. npf has a `roll_dates` of its own in `books/treasury_seasonal.py`, which + is a different function (a threshold on the back-adjustment offset, not a + `Delivery Month` change) and is unaffected. The `Delivery Month` column it read is still + in marketdata's frames, so the two-line derivation is available to anyone who wants it. + + **What stayed, deliberately:** the **databento** provider and the store-level + `write_prices` / `read_prices` it writes through. It has no marketdata equivalent yet, and + deleting it would destroy a validated provider-different alternative (ADR-0006) plus the + only intraday-capable source in the fleet. `cotdata-prices` therefore still exists, + scoped to `--ingest-databento` / `--build-databento`. There is no consumer *bar API* here + any more — read that output with `cotdata.store.read_prices(symbol, adjustment)`. + + The `metadata` manifest domain is still declared but has no writer, so pre-0.4.0 stores + can still migrate and reconcile their existing entries rather than stranding them in the + legacy aggregate. + + **Known breakage, recorded rather than fixed:** `crowdmon` (frozen, archived) and + `npf/docs/crowdmon/reproduce_forced_flow_mechanism.py` still call `cotdata.get_prices`. + Both are point-in-time records under their repos' doc lifecycle and were left untouched + on purpose. + ### Added - **The CFTC Supplemental (Commodity Index Trader) report, as a fourth `report_type`.** 13 select agricultural markets with the index-trader book split out of the commercial and diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 8b34fb7..24fbd3b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -75,9 +75,14 @@ pytest tests/ ``` ### Important Note on Norgate Testing -The Norgate integration (`src/cotdata/providers/norgate.py` and `tests/test_adjustment.py`) **only runs on Windows** because it requires the proprietary Norgate Data Updater application to be installed, running, and authenticated locally. -- If you submit a PR on a Mac or Linux machine, only the generic CFTC parsing tests will run locally. -- The CI pipeline will attempt to run cross-platform tests where possible, but cannot test live Norgate API calls. Please be extremely careful when modifying Norgate adapter logic. +There is none here any more. ADR-0007 moved the Norgate integration to +[`crucible-marketdata`](https://github.com/mspinola/marketdata), so Norgate changes and +their Windows-only constraints belong in that repo's CONTRIBUTING. + +Everything in this repo runs cross-platform with no vendor SDK: the CFTC parsers hit +cftc.gov, and the databento provider's tests drive a synthetic raw store rather than the +API. If a test needs a network or a paid key to pass, it does not belong in `tests/` — +put it in `scripts/` and say so in its docstring, as the databento parity harnesses do. ## Code Style diff --git a/README.md b/README.md index 79fedcb..9c823ee 100644 --- a/README.md +++ b/README.md @@ -5,26 +5,36 @@ [![Python versions](https://img.shields.io/pypi/pyversions/cotdata.svg)](https://pypi.org/project/cotdata/) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) -**A local, file-based data layer for futures prices and CFTC Commitments of Traders (COT) positioning.** +**A local, file-based data layer for CFTC Commitments of Traders (COT) positioning.** cotdata separates *fetching* data (a "producer" that talks to vendors) from *using* it (any number of "consumers" that just read Parquet through a small, stable API). Point every tool at one synced store, and none of them ever call a vendor SDK at runtime — so the same data feeds your research, backtests, and dashboards identically, on any OS. - **One store, many readers.** Consumers `import cotdata` and read; they never touch a vendor SDK. Swapping a data vendor is a producer-only change. -- **Free COT, optional paid prices.** CFTC Commitments of Traders data (1986–present) downloads free from cftc.gov on any OS. Futures prices/specs come from [Norgate](https://norgatedata.com/) (paid, Windows) and are optional. -- **Cross-platform reads.** Produce on Windows (for Norgate); read anywhere (Mac/Linux/Windows), offline. +- **Free, on any OS.** CFTC Commitments of Traders data (1986–present) downloads free from cftc.gov. No account, no subscription. - **Predecessor stitching.** `get_cot()` transparently stitches migrated CFTC codes (e.g. the Russell 2000) and rescales tick-size changes (e.g. Lumber) into one continuous series. - **Atomic writes.** Read the store safely even while the producer is downloading and writing. - **New-data signal.** Every run writes a structured `status.json` so downstream tools can poll one file to detect fresh data. +> [!IMPORTANT] +> **Price bars moved out of this package.** Under [ADR-0007](#ecosystem), cotdata is CFTC +> positioning only, and every daily bar — Norgate futures, Yahoo equities/ETFs, the +> `backadj`/`unadj`/`propadj` tiers, contract specifications — now lives in +> [**crucible-marketdata**](https://pypi.org/project/crucible-marketdata/). Replace +> `cotdata.get_prices(sym, adjustment)` with `marketdata.get_bars(sym, tier)` and point +> `MARKETDATA_STORE` at that store. The two packages keep separate stores and separate +> producers; nothing about `get_cot()` changed. +> +> The one price producer still here is **databento**, which has no marketdata equivalent +> yet (see [Cross-platform prices without Norgate](#cross-platform-prices-without-norgate-databento)). + ## Data sources at a glance | Data | Source | Cost | Runs on | |------|--------|------|---------| -| CFTC COT — legacy / disaggregated / TFF | [cftc.gov](https://www.cftc.gov/) | **Free** | any OS | -| Futures prices + contract specs | [Norgate Data](https://norgatedata.com/) | Paid subscription | **Windows** (producer only) | +| CFTC COT — legacy / disaggregated / TFF / supplemental | [cftc.gov](https://www.cftc.gov/) | **Free** | any OS | | Futures prices, back-adjusted | [Databento](https://databento.com/) GLBX.MDP3 | Paid per query | any OS | -| Futures prices, research-grade fallback | Yahoo Finance | **Free** | any OS | -| *Reading the store* (any of the above) | — | Free | any OS | +| *Reading the store* | — | Free | any OS | +| Daily bars + contract specs | → [crucible-marketdata](https://pypi.org/project/crucible-marketdata/) | — | — | ## Contents @@ -51,7 +61,7 @@ Report_Date_as_MM_DD_YYYY 2026-07-07 1969636 1435736 1502199 244103 286994 ``` -Futures **prices** additionally require a Norgate subscription on Windows — see [Producing data](#producing-data-producer). +For daily **bars**, install [crucible-marketdata](https://pypi.org/project/crucible-marketdata/) alongside this and read `marketdata.get_bars("ES", "backadj")`. ## How it works @@ -59,15 +69,15 @@ The **store is the API boundary** — not Python imports. Producers write Parque ``` PRODUCER — runs where each source is reachable - Norgate export (Windows) CFTC COT download (any OS) + CFTC COT download (any OS) databento ingest/build (any OS, paid) │ │ └──────────────┬───────────────┘ ▼ write parquet + manifest ┌────────────────────────────────────────────────────────────┐ │ CANONICAL STORE ($COTDATA_STORE) │ - │ prices/ cot_legacy/ cot_disagg/ cot_tff/ │ - │ cot_supplemental/ │ - │ metadata/ manifest.json status.json │ + │ cot_legacy/ cot_disagg/ cot_tff/ │ + │ cot_supplemental/ prices/ (databento only) │ + │ manifests/ status.json │ └────────────────────────────────────────────────────────────┘ │ read (offline, any OS) ┌──────────────┴───────────────┐ @@ -75,17 +85,19 @@ The **store is the API boundary** — not Python imports. Producers write Parque your signal research your backtest / dashboards both just: import cotdata · store synced via rsync / Dropbox / S3 + + Daily bars live in a SEPARATE store: $MARKETDATA_STORE, read with + `import marketdata`. Two stores, two producers, one seam (ADR-0007). ``` The store layout: -- `prices/{symbol}_{adjustment}.parquet` — Open/High/Low/Close/Volume/Open Interest, tz-naive `Date` index. `adjustment` ∈ {`backadj`, `unadj`} on disk; `propadj` is a third view **derived on read** (not stored). Close = exchange settlement. - `cot_legacy/{symbol}_{code}.parquet` — weekly CFTC Legacy positioning. - `cot_disagg/{symbol}_{code}.parquet` — weekly CFTC Disaggregated positioning. - `cot_tff/{symbol}_{code}.parquet` — weekly CFTC Traders in Financial Futures positioning. - `cot_supplemental/{symbol}_{code}.parquet` — weekly CFTC Supplemental (Commodity Index Trader) positioning, 13 agricultural markets. **Futures-and-options combined**, unlike the three above. -- `metadata/contract_specs.parquet` — Norgate contract specifications (tick size, point value, margin). -- `manifest.json` — per-table `last_date`, `n_rows`, `source`, `updated_at`, `schema_version`. +- `prices/{symbol}_{adjustment}.parquet` — **databento-built bars only**, `adjustment` ∈ {`backadj`, `unadj`}. This is the ADR-0006 alternative producer's output, not a general bar store: read Norgate/Yahoo bars from `marketdata` instead. +- `manifests/{cot,prices}.json` — per-table `last_date`, `n_rows`, `source`, `updated_at`, `schema_version`, one file per producer half. - `status.json` — machine-readable new-data signal for downstream tools (see [Operations](#operations)). - `vintage/` — optional as-published (vintage) capture: retained raw CFTC downloads plus change-only observations and field-level revisions. Purely additive; the tables above are @@ -98,11 +110,6 @@ Set `COTDATA_STORE` to the synced store directory, then: ```python import cotdata -# Prices — pick the adjustment that matches your use: -signals = cotdata.get_prices("ES", adjustment="backadj") # signals + stops (gap-free rolls) -sizing = cotdata.get_prices("ES", adjustment="unadj") # position sizing (true dollar prices) -milk = cotdata.get_prices("DC", adjustment="propadj") # ratio-adjusted: strictly positive, %-return preserving - # COT — four CFTC report families: legacy = cotdata.get_cot("ES", report="legacy") # Commercial / Non-Commercial disagg = cotdata.get_cot("ES", report="disagg") # Managed Money, Swap Dealers, ... (commodities) @@ -115,7 +122,15 @@ cit = cotdata.get_cot("CC", report="supplemental") # Index Traders (13 ags, > its `Open_Interest_All` is a different quantity for the same market and week. Do not > difference or ratio across reports without accounting for that. -A price frame (`get_prices("ES", adjustment="backadj").tail(3)`): +Daily **bars** come from the sibling package, against its own store: + +```python +import marketdata # pip install crucible-marketdata + +signals = marketdata.get_bars("ES", "backadj") # signals + stops (gap-free rolls) +sizing = marketdata.get_bars("ES", "unadj") # position sizing (true dollar prices) +milk = marketdata.get_bars("DC", "propadj") # ratio-adjusted, %-return preserving +``` ``` Open High Low Close Volume Open Interest @@ -125,16 +140,18 @@ Date 2026-07-14 7557.00 7613.75 7531.50 7591.25 1139735.0 0.0 ``` +Same frames, same symbols, same tier names — the import and the environment variable +(`MARKETDATA_STORE`) are what changed. `cotdata.get_prices` and `cotdata.roll_dates` are +gone rather than deprecated: left importable they would read a store the nightly job no +longer fills, and stale data is harder to notice than an `AttributeError`. + **Predecessor stitching & scaling:** `get_cot()` doesn't just read a file — it stitches historical CFTC codes for contracts that migrated exchanges (e.g. the Russell 2000) and rescales data for contracts that changed tick sizes (e.g. Lumber), so downstream models see one clean, continuous asset. ## Producing data (producer) -Run on the machine that can reach the source. Norgate prices require Windows, CFTC COT runs anywhere, and a server without Norgate can build prices from Databento instead (see [Cross-platform prices without Norgate](#cross-platform-prices-without-norgate-databento)). +Run on the machine that can reach the source. CFTC COT runs anywhere; the optional Databento price build also runs anywhere (see [Cross-platform prices without Norgate](#cross-platform-prices-without-norgate-databento)). Norgate and Yahoo bars are produced by `marketdata-update --bars`, in the [sibling package](https://pypi.org/project/crucible-marketdata/). ```bash -COTDATA_STORE=/store cotdata-update --prices # Norgate prices, ALL registry symbols (Windows) -COTDATA_STORE=/store cotdata-update --prices --symbols ES NQ # ...or a subset -COTDATA_STORE=/store cotdata-update --metadata # Norgate contract specs (Windows) COTDATA_STORE=/store cotdata-update --cot-legacy # CFTC Legacy (any OS) COTDATA_STORE=/store cotdata-update --cot-disagg # CFTC Disaggregated (any OS) COTDATA_STORE=/store cotdata-update --cot-tff # CFTC Traders in Financial Futures (any OS) @@ -143,25 +160,16 @@ COTDATA_STORE=/store cotdata-update --cot-all # all four CFTC COTDATA_STORE=/store cotdata-vintage fetch # optional: capture as-published COT (any OS) ``` -`--prices` with no `--symbols` updates every symbol in the registry; add `--symbols` to scope it. Each run prints a per-symbol line with the date advance (e.g. `ES: … [2026-07-13 -> 2026-07-14]`) and a summary footer (OK/failed counts, rows written, elapsed, newest date). A run **exits non-zero** if a fetch hard-fails (Norgate/CFTC unreachable), so a scheduler can retry — see [Scheduling on Windows](#scheduling-on-windows-task-scheduler). - -### Installation for the producer - -```bash -pip install "cotdata[norgate]" # adds the norgatedata dependency (Windows) -``` - -The `norgatedata` package talks locally to the Norgate Data Updater application — there are no API keys. You just need the Updater installed, authenticated, and running. +Each run prints a per-domain line and a summary footer. A run **exits non-zero** if a fetch hard-fails (CFTC or Databento unreachable), so a scheduler can retry — see [Scheduling on Linux](docs/LINUX_SCHEDULING.md). ### Cross-platform prices without Norgate (databento) -A server that cannot run Norgate (for example the public dashboard host) can build the price store from Databento instead. One provider owns each symbol end to end, so this is a full replacement, not a blend. Install the extras and set the environment: +A server that cannot run Norgate (for example the public dashboard host) can build a price store from Databento instead. One provider owns each symbol end to end, so this is a full replacement, not a blend. Install the extra and set the environment: ```bash -pip install "cotdata[databento,yahoo]" # databento producer + the Yahoo fallback +pip install "cotdata[databento]" export COTDATA_STORE=/path/to/store # the store the dashboard reads -export COTDATA_PRICE_SOURCE=databento # deployment default, so softs/MSCI fall to Yahoo export DATABENTO_API_KEY=db-... # for the paid ingest step only # optional: export COTDATA_DATABENTO_RAW=/path/to/raw # defaults to $COTDATA_STORE/_raw/databento ``` @@ -171,26 +179,33 @@ Then build the store in order (ingest before build): ```bash cotdata-update --ingest-databento # Stage 1 (PAID): raw .n.0/.n.1 ohlcv-1d + statistics -> raw store cotdata-update --build-databento # Stage 2 (FREE): additive back-adjustment -> $COTDATA_STORE/prices -cotdata-update --prices-yahoo # softs, lumber, MSCI proxies (resolve to Yahoo on this deployment) cotdata-update --cot-all # CFTC COT, the dashboard needs it too cotdata-update --check # coverage, newest dates, staleness ``` - **Two stages, one paid.** Stage 1 is the only step that hits the API. It writes an append-only raw store and resumes from the last fetched date, so re-runs pull only new days. Stage 2 reads that raw store with no API cost, so the back-adjustment can be iterated offline. The raw store is producer-internal, so keep it out of any sync to consumers. -- **History starts 2010-06-06** (the GLBX floor), shallower than Norgate. Markets not on CME Globex (ICE softs, lumber, MSCI intl) fall back to Yahoo. +- **History starts 2010-06-06** (the GLBX floor), shallower than Norgate. Markets not on CME Globex (ICE softs, lumber, MSCI intl) are not covered — take those from `marketdata`, which prices them off Yahoo. - **First-run check.** A healthy symbol prints `built unadj+backadj (N bars, K rolls)`. If it prints `no rolls detected`, back-adjustment is a no-op for that symbol, so investigate before trusting it. -- **Validate against Norgate** (optional gate) with `scripts/validate_databento_vs_norgate.py` if you have both stores. +- **Read it back** with `cotdata.store.read_prices(symbol, adjustment)`. There is no consumer bar API here — this store is the alternative producer's output, and the general one is `marketdata.get_bars`. +- **Validate against Norgate** (optional gate) with `scripts/validate_databento_vs_norgate.py`, pointing `--norgate-store` at a `$MARKETDATA_STORE`. - **Schedule** the two price commands nightly and `--cot-all` weekly — see [Scheduling on Linux](docs/LINUX_SCHEDULING.md). +> [!NOTE] +> Databento staying here is a deliberate exception to ADR-0007, not an oversight. It has +> no marketdata equivalent yet, and deleting it would destroy a validated alternative +> producer (ADR-0006) plus the only intraday-capable source in the fleet. When it is +> ported, `prices/`, `store.write_prices`/`read_prices` and the `prices` manifest half +> go with it and this package becomes COT-only in full. + ### Producer halves: one host, one job -`cotdata` has two producers by design: the CFTC downloader (free, any OS) and the price -producer (Norgate needs Windows). Two entry points scope a host to one of them: +`cotdata` has two producers by design: the CFTC downloader and the databento price +builder. Two entry points scope a host to one of them: ```bash -cotdata-cot --cot-all # CFTC half, any OS -cotdata-prices --prices --metadata --require-final # price half, Windows for Norgate -cotdata-update ... # both, for a single-machine deployment +cotdata-cot --cot-all # CFTC half +cotdata-prices --ingest-databento --build-databento # price half +cotdata-update ... # both, for a single-machine deployment ``` Each scoped entry point refuses the other half's flags, so a price box cannot quietly @@ -215,21 +230,24 @@ once every consumer of that store is on this version. See ADR-0007. ### Scheduling on Windows (Task Scheduler) -Full setup, including wrapper scripts, the three-task layout (daily prices, daily COT catch-up, Friday release-window poller), `--require-final` event-driven pricing, restart-on-failure retry settings, and Norgate/Task-Scheduler troubleshooting (notably: NDU needs an interactive session), is in **[docs/WINDOWS_SCHEDULING.md](docs/WINDOWS_SCHEDULING.md)**. Start with the [Windows Setup Guide](docs/WINDOWS_SETUP.md) first if Python/the venv/`COTDATA_STORE` aren't configured yet. +The COT tasks (daily catch-up, Friday release-window poller), restart-on-failure retry settings, and Task-Scheduler troubleshooting are in **[docs/WINDOWS_SCHEDULING.md](docs/WINDOWS_SCHEDULING.md)**. Start with the [Windows Setup Guide](docs/WINDOWS_SETUP.md) first if Python/the venv/`COTDATA_STORE` aren't configured yet. -The short version: prices fire once daily near the Norgate Continuous Futures Final (~8:55pm ET), COT gets a daily morning catch-up plus a tight Friday-afternoon poll around its ~3:30pm ET release, and every task uses restart-on-failure so idempotent, cheap re-runs absorb both transient errors and "not published yet." +The short version: COT gets a daily morning catch-up plus a tight Friday-afternoon poll around its ~3:30pm ET release, and every task uses restart-on-failure so idempotent, cheap re-runs absorb both transient errors and "not published yet." + +The Windows box is also where the **Norgate bar** job runs, but that is now `marketdata-update --bars --domain futures --require-final` from the sibling package, on its own schedule near the Norgate Continuous Futures Final (~8:55pm ET). Both packages' docs cover their own half. ### Scheduling on Linux (cron) -Full setup, including wrapper scripts, the crontab entries (nightly prices, daily COT catch-up, Friday release-window poller), `flock` overlap protection, and troubleshooting (cron's bare environment, timezone conversion, `DATABENTO_API_KEY` not being picked up), is in **[docs/LINUX_SCHEDULING.md](docs/LINUX_SCHEDULING.md)**. +Full setup, including wrapper scripts, the crontab entries (nightly databento prices, daily COT catch-up, Friday release-window poller), `flock` overlap protection, and troubleshooting (cron's bare environment, timezone conversion, `DATABENTO_API_KEY` not being picked up), is in **[docs/LINUX_SCHEDULING.md](docs/LINUX_SCHEDULING.md)**. -The short version: a databento server schedules the same way as the Windows/Norgate producer — prices nightly, COT gets a daily morning catch-up plus a tight Friday-afternoon poll around its ~3:30pm ET release, all idempotent and safe to over-run. +The short version: a databento server schedules prices nightly, and COT gets a daily morning catch-up plus a tight Friday-afternoon poll around its ~3:30pm ET release, all idempotent and safe to over-run. ### Syncing the store between machines -Norgate needs Windows, so a research Mac or a Linux dashboard is usually a **read-only -replica** of a store produced elsewhere. Prefer one producer writing everything and a -strictly one-directional sync. +A research Mac or a Linux dashboard is usually a **read-only replica** of a store +produced elsewhere. Prefer one producer writing everything and a strictly +one-directional sync. (The bar store syncs the same way, separately — it is a different +directory with a different producer.) Two directories must be **excluded** for size: `_cache/` (cotdata's cache of downloaded CFTC source zips) and `_raw/` (the **paid** databento raw store) are producer-internal @@ -299,25 +317,27 @@ COT tables are stored per code as **`{symbol}_{code}`** (e.g. `RTY_23977A`), so ### Back-adjusted vs unadjusted prices -Futures contracts expire, forcing traders to "roll" into the next contract, which usually trades at a slightly different price. Simply stitching contracts together creates artificial price gaps, so cotdata stores two series and derives a third: +Futures contracts expire, forcing traders to "roll" into the next contract, which usually trades at a slightly different price. Simply stitching contracts together creates artificial price gaps, so two series are stored and a third is derived: - **`backadj` (signals & stops).** Gap-free *arithmetic* (additive) rolls shift historical prices to align with the new contract, preserving *absolute* daily point moves. Use this for indicators, signals, and stop-losses to avoid false triggers on rollover gaps. - **`unadj` (position sizing).** Back-adjustment shifts historical prices (sometimes negative), so you can't use it for dollar values. Use `unadj` (raw, real-life prices) for that day to compute true dollar risk and contract counts. -- **`propadj` (proportional / ratio adjustment — strictly positive).** Derived on read from `unadj` + `backadj`; preserves daily *percentage* returns and never goes non-positive. Use it for **low-priced, long-history contracts where additive back-adjustment accumulates roll gaps below zero** and breaks price-based stops and R-multiples. See *Class III Milk (DC)* below. +- **`propadj` (proportional / ratio adjustment).** Derived on read from `unadj` + `backadj`; preserves daily *percentage* returns. Use it for **low-priced, long-history contracts where additive back-adjustment accumulates roll gaps below zero** and breaks price-based stops and R-multiples — see *Class III Milk (DC)* below. + +The databento builder here writes `backadj` and `unadj`. The tier *reader*, including the +derived `propadj`, is `marketdata.get_bars(symbol, tier)` — see that package's README for +the full treatment. Kept here because it is what the databento build produces and how to +judge whether it produced it correctly. #### Why `propadj` exists — Class III Milk (DC) Norgate publishes continuous futures in only two forms: unadjusted and **additive** back-adjusted (`_CCB`) — there is no native ratio-adjusted series. Additive adjustment subtracts each roll's calendar spread from all prior history, and for a low-priced, seasonal, ~29-year contract like **DC (Class III Milk, ~$15–20/cwt)** those gaps accumulate past zero: **46.7% of `DC_backadj` closes are ≤ 0** (range −9.83 to 23.09). A price-based stop, an R-multiple, or a percentage return is meaningless on a non-positive series, so CMR cannot use DC's `backadj` at all — even though DC is the flagship *new-asset-class* (Dairy) held-out generalization market. -`propadj` salvages it. Because the additive series `B` and unadjusted series `U` differ by an offset `O = B − U` that steps only at rolls, each roll's calendar spread is recoverable (`s = O[r−1] − O[r]`) and convertible to a multiplicative roll ratio `k = (U[r−1] + s)/U[r−1]`. Scaling each historical segment by the cumulative product of `k` (most-recent segment anchored to actual prices) yields a series that is **strictly positive over the full 1997–2026 history** (DC range 4.68–25.01), preserves within-segment percentage returns exactly, and is sign-identical to `backadj` on every day including rolls. It is a pure function of two already-stored series, so it needs no producer re-run — `get_prices("DC", adjustment="propadj")` works today. Recommendation: **CMR reads DC (and any similarly low-priced contract) with `adjustment="propadj"`.** Restricting DC to its positive-price era (2011→present, ~15y) or dropping it were the fallbacks; neither is needed. +`propadj` salvages it. Because the additive series `B` and unadjusted series `U` differ by an offset `O = B − U` that steps only at rolls, each roll's calendar spread is recoverable (`s = O[r−1] − O[r]`) and convertible to a multiplicative roll ratio `k = (U[r−1] + s)/U[r−1]`. Scaling each historical segment by the cumulative product of `k` (most-recent segment anchored to actual prices) preserves within-segment percentage returns exactly and is sign-identical to `backadj` on every day including rolls. Note it is **not** strictly positive, as this section once claimed: ratio adjustment scales by a positive factor, so it preserves the underlying's sign — CL's `propadj` close on 2020-04-20 is −24.11 because WTI really settled at −37.63. That is one bar in the whole store, against `backadj`'s 46.7% for DC. Recommendation: **read DC (and any similarly low-priced contract) with `marketdata.get_bars(sym, "propadj")`.** ### Providers & authentication -Which vendor prices a symbol is a deployment choice, not a fixed fact (the same ES is Norgate for local research and databento on a public-dash server). It is resolved when a producer runs, from three inputs: the deployment default `COTDATA_PRICE_SOURCE` (`norgate` if unset), per-symbol capability (the `norgate` / `databento` / `yahoo` mappings in the registry, `null` where a vendor has no series), and an optional per-symbol `price_source` override. A symbol uses its override if set, otherwise the default when that vendor can serve it, otherwise a Yahoo fallback where a ticker exists. Each producer writes only the symbols that resolve to it, so a symbol is never blended across vendors. One provider owns each symbol end to end. - -- **Norgate Data (paid, Windows).** No Python API key. The `norgatedata` package talks locally to the Norgate Data Updater app, which must be installed, authenticated, and running on Windows. This is the default for local research (`cotdata-update --prices`). -- **Databento (paid, cross-platform).** A two-stage producer for a server that cannot run Norgate. Stage 1 (`cotdata-update --ingest-databento`) pulls raw `.n.0` / `.n.1` `ohlcv-1d` and `statistics` into an append-only raw store (`$COTDATA_DATABENTO_RAW`, else `_raw/databento` under the store). This is the only paid step, and it is resumable, so re-runs fetch only new dates. Stage 2 (`cotdata-update --build-databento`) derives the back-adjusted prices from the raw store with no API cost, so the build logic can be iterated offline. Set `DATABENTO_API_KEY` and `COTDATA_PRICE_SOURCE=databento`. History starts 2010-06-06, and markets not on CME Globex (ICE softs, lumber, MSCI intl) fall back to Yahoo. The raw store is producer-internal, so exclude it from any consumer sync. -- **Yahoo Finance (free, research-grade).** `cotdata-update --prices-yahoo` prices the markets that resolve to yfinance on this deployment: the MSCI ETF proxies always, plus the softs and lumber on a databento server. Expect gaps and silent revisions. This is not a production replacement for the paid feeds. Requires the `[yahoo]` extra. +- **Databento (paid, cross-platform).** A two-stage producer for a server that cannot run Norgate. Stage 1 (`cotdata-update --ingest-databento`) pulls raw `.n.0` / `.n.1` `ohlcv-1d` and `statistics` into an append-only raw store (`$COTDATA_DATABENTO_RAW`, else `_raw/databento` under the store). This is the only paid step, and it is resumable, so re-runs fetch only new dates. Stage 2 (`cotdata-update --build-databento`) derives the back-adjusted prices from the raw store with no API cost, so the build logic can be iterated offline. Set `DATABENTO_API_KEY`. History starts 2010-06-06, and markets not on CME Globex (ICE softs, lumber, MSCI intl) are not covered. The raw store is producer-internal, so exclude it from any consumer sync. +- **Norgate Data / Yahoo Finance** — moved to [crucible-marketdata](https://pypi.org/project/crucible-marketdata/) with the rest of the bar production (ADR-0007), along with the `[norgate]` and `[yahoo]` extras and the `COTDATA_PRICE_SOURCE` resolution that chose between vendors. That per-symbol resolution now lives in marketdata's registry. ### The symbol registry @@ -328,7 +348,7 @@ The supported futures contracts are defined in a YAML registry, so adding a mark ### Atomic store -The store uses **atomic writes** (write-temp-then-rename). Consumers can safely query via `get_prices` / `get_cot` even while `cotdata-update` is actively downloading and writing. +The store uses **atomic writes** (write-temp-then-rename). Consumers can safely query via `get_cot` even while `cotdata-update` is actively downloading and writing. ### COT vintage tracking (as-published history) @@ -432,18 +452,24 @@ export COTDATA_STORE=/path/to/synced/store # the shared store uv run pytest # run the tests ``` -On the Windows producer, install the Norgate extra with `uv pip install -e ".[norgate]"` (tested on Python 3.10, within Norgate's supported versions). Use `uv run `, or activate with `source .venv/bin/activate` (Mac/Linux) / `.venv\Scripts\activate` (Windows). +For the databento producer, add the extra with `uv pip install -e ".[databento]"`. Use `uv run `, or activate with `source .venv/bin/activate` (Mac/Linux) / `.venv\Scripts\activate` (Windows). ## Reference: Data schemas The canonical store uses standard Parquet files. Loaded with `pd.read_parquet()`, they conform to the following schemas. ### Price Data (`prices/{symbol}_{adjustment}.parquet`) -Primary price history (Norgate Data), indexed by tz-naive `Date`. The pipeline downloads both the back-adjusted (`backadj`) series for signals/stops and the unadjusted (`unadj`) series for true transaction-cost modeling. -**Reading reconstructed volume:** the reconstruction columns below are internal storage. Consumers should not read `Volume_Reconstructed` directly — call `get_prices(symbol, volume="reconstructed")` and the `Volume` column is served as reconstructed-with-per-row-raw-fallback, plus a `Volume_Source` column for audit. The default `volume="front"` returns the front-month series unchanged (byte-identical to the pre-v2 API). +> [!NOTE] +> **Databento output only.** Norgate/Yahoo bars and the general bar schema (including the +> reconstructed-volume columns and how to read them) moved to +> [crucible-marketdata](https://pypi.org/project/crucible-marketdata/) under ADR-0007. +> What is documented here is what `--build-databento` writes into this store, read back +> with `cotdata.store.read_prices(symbol, adjustment)`. + +Indexed by tz-naive `Date`. The build writes both the back-adjusted (`backadj`) series for signals/stops and the unadjusted (`unadj`) series for true transaction-cost modeling. -**Schema versioning:** `schema_version` in `manifest.json` records the on-disk data version (v2 = reconstructed volume promoted). Consumers key cache invalidation on `cotdata.schema_version()` and can guard with `cotdata.require_schema(min_version)`. +**Schema versioning:** `schema_version` in the manifest records the on-disk data version. Consumers key cache invalidation on `cotdata.schema_version()` and can guard with `cotdata.require_schema(min_version)`. | Column | Type | Description | |--------|------|-------------| @@ -453,29 +479,14 @@ Primary price history (Norgate Data), indexed by tz-naive `Date`. The pipeline d | `Low` | float | Low price. | | `Close` | float | Settlement Close price. | | `Volume` | float | Continuous contract trading volume (front-month only). | -| `Open Interest` | float | Continuous contract open interest. | -| `Volume_Reconstructed` | float | True market volume (sum of First and Second contract). Differs from raw `Volume` by symbol — typically higher for products whose rolls spread volume across contracts, but roughly equal or lower for symbols with a near-empty back month (e.g. crypto). Not a drop-in replacement. | -| `Volume_Source` | string | `reconstructed` if First+Second available, `raw` fallback if not. | -| `FirstVolume` / `SecondVolume` | float | Trading volume of the specific first and second expiring contracts. | -| `FirstContract` / `SecondContract` | string | Contract names for the first and second expirations (e.g., `ES-2024H`). | +| `Open Interest` | float | Continuous contract open interest, from the `statistics` schema (stat_type 9). | | `Delivery Month` | float | Expiration month of the active contract (e.g. `202609`). Used to detect contract rolls. | -### Contract Specifications (`metadata/contract_specs.parquet`) -Contract metadata (Norgate Data), used for exact point-value risk sizing and transaction cost models. +### Contract Specifications -| Column | Type | Description | -|--------|------|-------------| -| `Symbol` | string | Internal ticker symbol (e.g., `ES`). | -| `Norgate_Symbol` | string | Raw Norgate symbol used to query the API (e.g., `&ES_CCB`). | -| `Name` | string | Full name of the contract. | -| `Exchange` | string | Name of the listing exchange. | -| `Group` | string | Norgate asset classification group. | -| `Contract Size` | float | Size multiplier (e.g., $50 for ES). Also called Point Value. | -| `Tick Size` | float | Minimum price fluctuation (e.g., 0.25 for ES). | -| `Tick Value` | float | Dollar value of one tick (`Tick Size` * `Contract Size`). | -| `Point Value` | float | Same as `Contract Size`. | -| `Currency` | string | Base currency of the contract. | -| `Margin` | float | Initial margin requirement (if provided by Norgate). | +Moved to `marketdata`'s store (`metadata/contract_specs.parquet` under +`$MARKETDATA_STORE`) with the Norgate producer that writes them — read them with +`marketdata.read_metadata()`. Nothing in this package writes contract specs any more. ### COT Legacy Data (`cot_legacy/{symbol}_{code}.parquet`) Legacy positioning data (CFTC Legacy Futures Report). **History starts in 1986.** Indexed by tz-naive `Report_Date_as_MM_DD_YYYY`. @@ -542,13 +553,12 @@ The CFTC publishes positioning data in four formats; `cotdata` manages all four ## Diagnostics -Verify your Norgate subscription and configuration with the included smoke test, on the Windows producer: - ```bash -python tests/test_adjustment.py +cotdata-update --check # coverage, newest dates, staleness — no network +python scripts/vintage_alert_selftest.py # the vintage capture's alerting path ``` -It checks: (1) **Local communication** — Python can reach the Norgate Data Updater; (2) **Subscription access** — your subscription includes the required CME futures package; (3) **Roll-gap validation** — proves whether the Updater is returning back-adjusted (gap-free) vs unadjusted continuous contracts, by hunting for calendar-spread gaps at roll dates. Gap-free data is vital for accurate stop-loss modeling. +Norgate subscription and roll-gap diagnostics moved to `marketdata` with the provider. ## Ecosystem @@ -556,8 +566,12 @@ cotdata is the *data* layer of a small, unbundled toolchain — it stops at "cle data behind a stable API" on purpose. What you do with that data is a separate, swappable step: -- **cotdata** *(this package)* — the *data* layer. One synced store of futures - prices and CFTC COT positioning; many readers, no vendor SDK at read time. +- **cotdata** *(this package)* — CFTC COT positioning. One synced store, many + readers, no vendor SDK at read time. +- **[crucible-marketdata](https://pypi.org/project/crucible-marketdata/)** — the + daily bars that used to live here: Norgate futures, Yahoo equities/ETFs, the + adjustment tiers, contract specs. Split out by ADR-0007 so a positioning question + and a price question have separate stores, producers and schedules. - **[crucible](https://github.com/mspinola/crucible)** — the *edge* layer. Feed a signal built on cotdata frames into crucible and it tells you — with a confidence interval and a p-value — whether the trade-level edge is real, before @@ -572,12 +586,12 @@ COT/futures research — crucible is just the most common thing to point at it n Want to contribute or work on cotdata locally? See [CONTRIBUTING.md](CONTRIBUTING.md) for: - Virtual environment setup with `uv` or standard `pip` - Running the test suite -- Platform-specific notes (Norgate is Windows-only; CFTC parsing runs anywhere) +- Platform-specific notes (CFTC parsing and the databento build run anywhere) - Code style guidelines ## Contributing -Issues and pull requests are welcome. Please see [CONTRIBUTING.md](CONTRIBUTING.md) for setup, tests, and conventions. When filing a bug, include your OS — Norgate features require Windows, while store reads and CFTC COT run anywhere. +Issues and pull requests are welcome. Please see [CONTRIBUTING.md](CONTRIBUTING.md) for setup, tests, and conventions. When filing a bug, include your OS. If it is about price bars, it probably belongs on [crucible-marketdata](https://github.com/mspinola/marketdata) instead. ## License diff --git a/docs/LINUX_SCHEDULING.md b/docs/LINUX_SCHEDULING.md index a262e01..662e565 100644 --- a/docs/LINUX_SCHEDULING.md +++ b/docs/LINUX_SCHEDULING.md @@ -4,7 +4,7 @@ For the cross-platform Databento producer path (no Norgate/Windows required) — ## Goal -A databento server schedules the same way as the Windows/Norgate producer: **prices nightly**, **COT caught soon after its Friday ~3:30pm ET release**, with a daily catch-up for holiday delays. The same two properties hold: +A databento server schedules **prices nightly** and **COT soon after its Friday ~3:30pm ET release**, with a daily catch-up for holiday delays. Two properties hold: - **Idempotent.** `--cot-all` HEAD-checks each CFTC year zip and skips it if unchanged. `--ingest-databento` resumes from the last fetched date, so a re-run pulls only new days. Running before new data lands is a harmless no-op. - **Fails loudly.** A run exits non-zero only on a hard fetch error (source unreachable), not when there is simply nothing new. Because ingest is resumable and COT is idempotent, a failed or missed run is picked up by the next one, so no explicit retry logic is needed. @@ -17,20 +17,24 @@ Cron runs with a bare environment, so put the config and the venv path in a wrap Inside the scripts, overwrite the plain-text markers: `REPLACE_WITH_STORE_PATH` = your store, `REPLACE_WITH_VENV_PATH` = your virtualenv, `REPLACE_WITH_DATABENTO_KEY` = your Databento key. (They're plain markers, not `<...>` placeholders, because an unedited `<...>` would be read as a shell redirection and the script would fail.) The `` in the crontab lines below is normal fill-in notation. -`run-prices.sh` — the two-stage databento build plus the Yahoo fallback: +`run-prices.sh` — the two-stage databento build: ```bash #!/usr/bin/env bash set -euo pipefail export COTDATA_STORE=REPLACE_WITH_STORE_PATH -export COTDATA_PRICE_SOURCE=databento export DATABENTO_API_KEY=REPLACE_WITH_DATABENTO_KEY BIN=REPLACE_WITH_VENV_PATH/bin/cotdata-prices "$BIN" --ingest-databento # Stage 1 (paid): raw .n.0/.n.1 to raw store "$BIN" --build-databento # Stage 2 (free): back-adjusted prices -"$BIN" --prices-yahoo # softs / lumber / MSCI fallback ``` +Databento is the only price producer left in this package. ADR-0007 moved the Norgate and +Yahoo bar producers to [`marketdata`](https://pypi.org/project/crucible-marketdata/), so the +markets databento does not cover — ICE softs, lumber, the MSCI ETF proxies — are fetched by +`marketdata-update --bars` against `$MARKETDATA_STORE` on whichever box produces that store, +not by this script. + `run-cot.sh` — COT (note the different command): ```bash diff --git a/docs/SYNCING.md b/docs/SYNCING.md index 917da22..cfb59f7 100644 --- a/docs/SYNCING.md +++ b/docs/SYNCING.md @@ -6,6 +6,15 @@ the producer and the consumers are different machines, something has to move the This page is about **what** to move and what to leave behind. The transport is the easy part and comes last. +> [!NOTE] +> **Two stores since ADR-0007.** Daily bars moved to +> [`crucible-marketdata`](https://pypi.org/project/crucible-marketdata/) and its own +> `$MARKETDATA_STORE`. The Windows box still produces both and still pushes both to the same +> replicas, so the topology, exclusions, auth gotchas and preflight advice below apply +> unchanged — there are now **two source directories to mirror instead of one**, and the +> `prices/`, `metadata/` rows in the exclusion table describe the bar store rather than this +> one. Where a command names `cotdata-prices --prices`, read +> `marketdata-update --bars --domain futures --require-final`. ## This deployment: one Norgate producer, two replicas A single Windows server is the only producer (Norgate prices, CFTC COT). It feeds two @@ -23,14 +32,14 @@ so nothing had to move. It is now a **standalone physical Windows server**, and folder was deliberately not carried over, so a real network sync replaced it. - **Producer:** the Windows server (Norgate prices, CFTC COT), one-directional. -- **Consumer:** the Mac, read-only, `COTDATA_PRICE_SOURCE` unset (Norgate default). +- **Consumer:** the Mac, read-only. - **Transport:** `robocopy /MIR` ([`examples/windows/sync-store.cmd`](examples/windows/sync-store.cmd)) pushing to an SMB share the Mac exports for `~/code/cotdata_store`, reached from the server as `\\\cotdata_store` (use the Mac's LAN IP if its name will not resolve from a headless server). - **Trigger:** chained onto the end of the producer task behind an `errorlevel` guard, so it fires only after a successful run rather than on a timer (a deferred - `--require-final` prices run exits non-zero and is skipped). + `--require-final` bar run exits non-zero and is skipped). - **Auth gotcha:** a task set to run whether the user is logged on or not has no cached SMB credentials in its non-interactive session, so the UNC write fails access-denied even though it works logged on. Store one on the server with `cmdkey /add`, or chain the @@ -46,7 +55,7 @@ synced Norgate: the dashboard then shows exactly what local research shows, at l maintenance than a per-symbol roll-rule table. - **Producer:** the same Windows server (Norgate prices, CFTC COT). -- **Consumer:** a remote Ubuntu VPS, read-only, `COTDATA_PRICE_SOURCE` unset (Norgate). +- **Consumer:** a remote Ubuntu VPS, read-only. - **Transport:** an `rsync` push over SSH, chained onto the producer task, using a packaged rsync on Windows (cwRsync or WSL). robocopy cannot speak SSH, and SMB must never be exposed over the internet, so the Mac's SMB path does not carry here. See @@ -94,18 +103,27 @@ interactive-session requirement. Give it its own task: | Task | Command | Task Scheduler "General" | |---|---|---| -| prices | `cotdata-prices --prices --metadata --require-final` | Run only when user is logged on (NDU needs a desktop session) | +| bars | `marketdata-update --bars --domain futures --require-final` | Run only when user is logged on (NDU needs a desktop session) | | COT | `cotdata-cot --cot-all` | Run whether user is logged in or not | +The bar task writes `$MARKETDATA_STORE` and the COT task writes `$COTDATA_STORE`. Two +producers writing two *disjoint* stores is not the two-producer hazard above — that one is +about two producers writing the *same* files. + ## What NOT to sync This is the part that matters, and on a real store it is most of the bytes. +Per store, since there are now two. `bars/` and `metadata/` live in `$MARKETDATA_STORE`; +everything else here is `$COTDATA_STORE`. `prices/` appears in the cotdata store only when +the databento producer is in use. + | Directory | Sync? | Why | |---|---|---| -| `prices/` | **yes** | the data | +| `bars/` (marketdata) | **yes** | the data | +| `metadata/` (marketdata) | **yes** | contract specs | | `cot_legacy/`, `cot_disagg/`, `cot_tff/` | **yes** | the data | -| `metadata/` | **yes** | contract specs | +| `prices/` | **yes**, if you run databento | the ADR-0006 alternative producer's output | | `manifests/` | **yes** | per-half bookkeeping | | `status.json` | yes | the producer's own view, useful on the replica | | `_cache/` | **NO** | cotdata's own download cache of CFTC source zips, producer-internal, free to rebuild | @@ -234,7 +252,7 @@ If a manifest arrives before the parquet it describes, a consumer briefly sees a pointing at data that has not landed. The other order is harmless: data present but not yet announced. -In practice this is a nicety rather than a hazard, because `get_prices` and `get_cot` +In practice this is a nicety rather than a hazard, because `get_cot` read parquet directly and the manifest is status only. But if your transport lets you control ordering, sync the data directories first and `manifests/` last. diff --git a/docs/WINDOWS_SCHEDULING.md b/docs/WINDOWS_SCHEDULING.md index 9484f3e..ff83ead 100644 --- a/docs/WINDOWS_SCHEDULING.md +++ b/docs/WINDOWS_SCHEDULING.md @@ -2,6 +2,19 @@ New to Python and cotdata on Windows? Start with the [Windows Setup Guide](WINDOWS_SETUP.md) — install Python, create the venv, and confirm `cotdata-update --cot-legacy` works by hand before automating it. +> [!IMPORTANT] +> **The price task now runs a different package.** ADR-0007 moved Norgate bar production to +> [`crucible-marketdata`](https://pypi.org/project/crucible-marketdata/), so the nightly job is +> `marketdata-update --bars --domain futures --require-final` against `$MARKETDATA_STORE`, not +> `cotdata-prices --prices`. It is still scheduled here, on the same box, at the same time, +> with the same restart-on-failure trick — every operational fact below about NDU, the +> interactive session, deferred exit codes and retry is unchanged. Only the command and the +> store variable moved. +> +> **Upgrading?** Edit `run-prices.cmd` to the new command and give it `MARKETDATA_STORE`. A +> wrapper still calling `cotdata-prices --prices` now dies on an unrecognised flag, which +> Task Scheduler shows as a failed run — loud, not silent. + ## Goal **Prices daily**, and **COT caught within minutes of its Friday ~3:30pm ET release** while surviving holiday delays. Two properties make this simple: @@ -11,18 +24,19 @@ New to Python and cotdata on Windows? Start with the [Windows Setup Guide](WINDO ## Wrapper scripts -Create **two** wrapper scripts — they run *different* commands. Each sets `COTDATA_STORE` and calls the venv's half-scoped entry point: `cotdata-prices` for the price half, `cotdata-cot` for the COT half. Each refuses the other half's flags, so a host is scoped to one job and a price box cannot quietly become a second COT producer. `cotdata-update` still runs both if a single machine must do everything. +Create **two** wrapper scripts — they run *different* commands from *different* packages: `marketdata-update` for the bars, `cotdata-cot` for the COT half. cotdata's own entry points stay half-scoped (`cotdata-cot` / `cotdata-prices`) and each refuses the other half's flags, so a host is scoped to one job and a price box cannot quietly become a second COT producer. > **Ready-made templates:** copy [`docs/examples/windows/run-prices.cmd`](examples/windows/run-prices.cmd) and [`run-cot.cmd`](examples/windows/run-cot.cmd) out of the repo into your `` (e.g. `C:\Users\you\cotdata\scheduler\`) rather than retyping them — then just fill in the placeholders. Keep them outside the repo so a `git pull` never clobbers your edited paths. > **Fill in your real paths.** Inside the `.cmd` files, overwrite the plain-text markers `REPLACE_WITH_STORE_PATH` (your synced store, e.g. `\\Mac\code\cotdata_store`) and `REPLACE_WITH_VENV_PATH` (your virtualenv, e.g. `C:\Users\you\code\cotdata\.venv`). **Don't use angle-bracket placeholders like `` inside a `.cmd`** — cmd reads `<` and `>` as redirection and the script fails with "The syntax of the command is incorrect," even on `REM` comment lines. The `` notation in the *task commands* further down is fine to substitute since those are quoted or typed at the prompt. -`run-prices.cmd` — prices (with `--require-final`, so it runs only once Norgate's **Final** prices are in, not interim bars): +`run-prices.cmd` — bars (with `--require-final`, so it runs only once Norgate's **Final** prices are in, not interim bars). Note `MARKETDATA_STORE`: a *different* directory from `COTDATA_STORE`, not an alias for it. ```bat @echo off -set COTDATA_STORE=REPLACE_WITH_STORE_PATH -"REPLACE_WITH_VENV_PATH\Scripts\cotdata-prices.exe" --prices --metadata --require-final +set MARKETDATA_STORE=REPLACE_WITH_MARKETDATA_STORE_PATH +"REPLACE_WITH_VENV_PATH\Scripts\marketdata-update.exe" --bars --domain futures --require-final +"REPLACE_WITH_VENV_PATH\Scripts\marketdata-update.exe" --metadata ``` `run-cot.cmd` — COT (note the **different** command, `--cot-all`): @@ -33,7 +47,7 @@ set COTDATA_STORE=REPLACE_WITH_STORE_PATH "REPLACE_WITH_VENV_PATH\Scripts\cotdata-cot.exe" --cot-all ``` -Using the full venv `\Scripts\cotdata-prices.exe` / `\Scripts\cotdata-cot.exe` path (rather than relying on the command being on `PATH`) matters here: Task Scheduler runs with a different, often bare, environment than your interactive shell, so a bare command name that resolves fine in Command Prompt can fail to resolve under the scheduler. +Using the full venv `\Scripts\marketdata-update.exe` / `\Scripts\cotdata-cot.exe` path (rather than relying on the command being on `PATH`) matters here: Task Scheduler runs with a different, often bare, environment than your interactive shell, so a bare command name that resolves fine in Command Prompt can fail to resolve under the scheduler. `run-vintage.cmd` — **optional**, the as-published (vintage) capture. Copy [`docs/examples/windows/run-vintage.cmd`](examples/windows/run-vintage.cmd); it runs @@ -53,9 +67,9 @@ before enabling it: Create three tasks (plus an optional fourth if you enable vintage capture) — times are the **machine's local** time; convert from ET if it isn't on Eastern: ```bat -:: 1) Prices — fire at the Continuous Futures Final (~8:55pm ET); --require-final + restart +:: 1) Bars — fire at the Continuous Futures Final (~8:55pm ET); --require-final + restart :: below keep retrying (cheap no-ops) until Norgate has actually pulled the Finals. -schtasks /Create /TN "cotdata prices" /TR "\run-prices.cmd" /SC DAILY /ST 20:55 +schtasks /Create /TN "marketdata bars" /TR "\run-prices.cmd" /SC DAILY /ST 20:55 :: 2) COT — daily morning catch-up for holiday-delayed releases and as a safety net schtasks /Create /TN "cotdata COT (catch-up)" /TR "\run-cot.cmd" /SC DAILY /ST 08:10 @@ -93,13 +107,13 @@ Register-ScheduledTask -TaskName "cotdata COT (Fri release)" -Action $act -Trigg (Or in the Task Scheduler GUI: New Task → Trigger *Weekly, Friday, 3:25pm* → check *"Repeat task every: 2 minutes for a duration of: 45 minutes."*) -**Event-driven prices with `--require-final`.** cotdata reads two Norgate databases: **Continuous Futures** (the `&ES` / `_CCB` series) and **Futures** (the individual `ES-2026H` contracts used to reconstruct volume). Their **Final** prices land ~8:40pm ET (Futures) and ~8:55pm ET (Continuous Futures), but your Norgate Data Updater still has to *pull* them on its next poll. Rather than guess a fixed time, `--require-final` checks `norgatedata.last_database_update_time()` for both databases and only fetches once each has been refreshed at/after `--final-cutoff` (default `20:55` local — set it to your machine's local equivalent of 8:55pm ET). Until then it **defers with a non-zero exit**, so the restart setting below turns "fire at 8:55pm" into "run the moment NDU has the Finals." +**Event-driven bars with `--require-final`.** The producer reads two Norgate databases: **Continuous Futures** (the `&ES` / `_CCB` series) and **Futures** (the individual `ES-2026H` contracts used to reconstruct volume). Their **Final** prices land ~8:40pm ET (Futures) and ~8:55pm ET (Continuous Futures), but your Norgate Data Updater still has to *pull* them on its next poll. Rather than guess a fixed time, `--require-final` asks a **data** question: does Norgate hold a NEWER settled bar than the store already does, for a quorum of liquid reference symbols? That needs no wall-clock cutoff and no trading calendar, so it is immune to Norgate's publish-time drift — an early publish is caught early and a late one simply defers. (It replaced a fixed `--final-cutoff`, which broke in production on 2026-07-27 when Norgate finalized one database at 8:49pm and the check demanded 8:55pm for both. See [design/finals_ready_data_driven.md](design/finals_ready_data_driven.md).) Until it is ready the run **defers with a non-zero exit** having fetched nothing, so the restart setting below turns "fire at 8:55pm" into "run the moment NDU has the Finals." **Retry / wait via restart-on-failure.** Give each task a *restart on failure* — it does double duty: it retries transient fetch errors, and (for the price task) waits out the gap between 8:55pm and NDU actually pulling the Finals (each retry is a cheap `last_database_update_time` check that exits immediately until ready). On a genuine no-session day the retries simply exhaust, harmlessly. `schtasks` can't set this, so use PowerShell (applies to all three tasks): ```powershell $s = New-ScheduledTaskSettingsSet -RestartInterval (New-TimeSpan -Minutes 10) -RestartCount 6 -foreach ($t in "cotdata prices","cotdata COT (Fri release)","cotdata COT (catch-up)") { +foreach ($t in "marketdata bars","cotdata COT (Fri release)","cotdata COT (catch-up)") { Set-ScheduledTask -TaskName $t -Settings $s } ``` @@ -117,7 +131,7 @@ Test in three layers: fire the task, read the result, then confirm it actually w Don't wait for the trigger — run it now: ```bat -schtasks /Run /TN "cotdata prices" +schtasks /Run /TN "marketdata bars" ``` (Or in `taskschd.msc`: right-click the task → **Run**.) Running the *task* rather than the `.cmd` by hand is the stronger test: it exercises the scheduler's own account, environment, and working directory, which is where scheduled runs usually differ from your interactive shell. @@ -125,7 +139,7 @@ schtasks /Run /TN "cotdata prices" ### 2. Read what happened ```bat -schtasks /Query /TN "cotdata prices" /V /FO LIST +schtasks /Query /TN "marketdata bars" /V /FO LIST ``` Check **Last Run Time** and **Last Result**. For per-run detail, enable history once (right-click the task or the library root → **Enable All Tasks History**) and read the task's **History** tab. @@ -142,12 +156,12 @@ Confirm the relevant `newest data` date advanced (and `last write (UTC)` is rece ### Testing the timing and conditions -- **A daytime prices run only proves the wrapper resolves** — with no Finals yet it just defers. To exercise the actual write path in daylight, run `cotdata-prices --prices --metadata` by hand (no `--require-final`), or fire the task after ~8:55pm ET. +- **A daytime bars run only proves the wrapper resolves** — with no Finals yet it just defers. To exercise the actual write path in daylight, run `marketdata-update --bars --domain futures` by hand (no `--require-final`), or fire the task after ~8:55pm ET. - **Test the trigger itself** by moving it a couple of minutes out, watching it fire, then setting it back: ```bat - schtasks /Change /TN "cotdata prices" /ST 14:20 + schtasks /Change /TN "marketdata bars" /ST 14:20 :: watch it run, then restore - schtasks /Change /TN "cotdata prices" /ST 20:55 + schtasks /Change /TN "marketdata bars" /ST 20:55 ``` This catches the two silent killers below — a disabled task, or the default *"only if on AC power"* condition skipping runs on a laptop. - **Keep a permanent record** by having the wrapper redirect output to a log file (see [Diagnosing a silent failure](#diagnosing-a-silent-failure)). @@ -196,19 +210,19 @@ visible, separate problem rather than a silent missing write. **The single most common cotdata-on-Task-Scheduler failure.** The `norgatedata` package doesn't call a remote API — it talks locally to the **Norgate Data Updater (NDU)** app, which is a GUI program that has to be running and authenticated *in your desktop session*. -If a task's General tab has **"Run whether user is logged in or not"** checked, Windows runs it in a non-interactive session (effectively no desktop), and it cannot reach an NDU instance running in your logged-in session — `cotdata-prices --prices` will fail even though NDU looks fine when you check it yourself. +If a task's General tab has **"Run whether user is logged in or not"** checked, Windows runs it in a non-interactive session (effectively no desktop), and it cannot reach an NDU instance running in your logged-in session — `marketdata-update --bars` will fail even though NDU looks fine when you check it yourself. **Fix:** for the prices task, use **"Run only when user is logged on"** (the default) so it executes in your interactive session alongside NDU. This does mean the machine needs to be logged in (not just powered on) at run time — screen lock is fine, logged-out is not. ### Task shows success but wrote nothing -`cotdata-update` exits non-zero on a hard fetch error, but a **deferred** `--require-final` run (NDU hasn't pulled the Finals yet) also exits non-zero — that's by design, not a bug, and the restart-on-failure setting is what turns those into a working poll loop. Don't "fix" this by making the wrapper swallow the exit code; that breaks retry. +`marketdata-update` exits non-zero on a hard fetch error, but a **deferred** `--require-final` run (NDU hasn't pulled the Finals yet) also exits non-zero — that's by design, not a bug, and the restart-on-failure setting is what turns those into a working poll loop. Don't "fix" this by making the wrapper swallow the exit code; that breaks retry. To confirm a run actually wrote data, check `status.json` in the store (`newest_data.prices` advancing) rather than trusting Task Scheduler's Last Run Result alone — see [Operations](../README.md#operations) in the README. -### Task Scheduler can't find `cotdata-prices` / `cotdata-cot` +### Task Scheduler can't find `marketdata-update` / `cotdata-cot` -Always call the fully-qualified `\Scripts\cotdata-prices.exe` (or `cotdata-cot.exe`) inside the wrapper `.cmd`, never a bare command name. The scheduler's environment doesn't inherit your interactive shell's activated venv or `PATH` changes. +Always call the fully-qualified `\Scripts\marketdata-update.exe` (or `cotdata-cot.exe`) inside the wrapper `.cmd`, never a bare command name. The scheduler's environment doesn't inherit your interactive shell's activated venv or `PATH` changes. ### Diagnosing a silent failure diff --git a/docs/WINDOWS_SETUP.md b/docs/WINDOWS_SETUP.md index e7823ee..f4d02e5 100644 --- a/docs/WINDOWS_SETUP.md +++ b/docs/WINDOWS_SETUP.md @@ -109,21 +109,27 @@ Run these from inside the clone (`C:\...\cotdata`, where you already are after S ### For Producer (Windows with Norgate) -If you have a Norgate subscription and will produce prices on this machine: +Norgate bars are produced by the **sibling package** since ADR-0007 — this repo no longer +talks to Norgate at all. On the Windows producer install both: + ```cmd -pip install -e ".[norgate]" -:: uv venv: uv pip install -e ".[norgate]" +pip install -e . :: cotdata (CFTC COT) +pip install "crucible-marketdata[norgate]" :: bars, incl. the norgatedata dependency +:: uv venv: uv pip install -e . && uv pip install "crucible-marketdata[norgate]" ``` +They keep separate stores (`COTDATA_STORE`, `MARKETDATA_STORE`) and separate schedules; the +rest of this guide covers both because they run on the same box. + ### For Consumer (Read-Only) -If you only read data (no Norgate producer), you do not need the clone at all: +If you only read data, you do not need the clone at all: ```cmd pip install cotdata :: uv venv: uv pip install cotdata ``` -From PyPI, current as of 0.3.0. If you already cloned and would rather track `main`, `pip install -e .` from inside the clone does the same job and updates on `git pull`. +From PyPI, current as of 0.4.0. If you already cloned and would rather track `main`, `pip install -e .` from inside the clone does the same job and updates on `git pull`. Installation may take 1–2 minutes (many dependencies). Watch for any errors — if it says a package failed to download, your internet may have glitched; try again. @@ -132,11 +138,14 @@ Verify installation: python -c "import cotdata; print(cotdata.__version__)" ``` -Should print a version number. **Note:** because the version hasn't been bumped, the editable clone install *and* the stale PyPI build both report `0.1.0` — the version string alone won't tell you which one you have. Confirm you got the right one by checking the CLI has the producer flags: +Should print a version number. Confirm the CLI resolves too: ```cmd cotdata-update --help ``` -If `--metadata` and `--require-final` appear, you're on the clone. If they don't, you installed the stale PyPI build — reinstall with `-e` from inside the clone. If `import` errors instead, your venv isn't activated (look for `(.venv)`/`(cotdata)` in the prompt). +If `--cot-all` and `--build-databento` appear, you are set. If `import` errors instead, your +venv isn't activated (look for `(.venv)`/`(cotdata)` in the prompt). If you see `--prices` or +`--metadata`, you are on a pre-0.4.0 build from before the ADR-0007 split — those flags are +gone, and Norgate bars come from `marketdata-update --bars`. ## Step 4: Set Environment Variables @@ -282,7 +291,7 @@ If you'd rather keep the `pip`-installed copy, add its `Scripts` directory to `P ### Dependencies Won't Install on a Newer Python -**Problem:** `pip install "cotdata[norgate]"` fails while building a dependency (`norgatedata`, or a package with no matching wheel), often with a compiler error or "no matching distribution found." Or `import cotdata` works but `cotdata-update` isn't found and `where python` / `where pip` point at an `AppData\Local\Python\pythoncore-3.14-64` path. +**Problem:** `pip install "crucible-marketdata[norgate]"` fails while building a dependency (`norgatedata`, or a package with no matching wheel), often with a compiler error or "no matching distribution found." Or `import cotdata` works but `cotdata-update` isn't found and `where python` / `where pip` point at an `AppData\Local\Python\pythoncore-3.14-64` path. **Cause:** You're on a Python newer than 3.11 (e.g. 3.13 or 3.14). The workspace pins **3.11** because that's what the dependencies publish prebuilt wheels for; on a brand-new Python, pip falls back to compiling from source (which usually fails) or installs into a global per-version location instead of your venv. @@ -295,7 +304,7 @@ If you'd rather keep the `pip`-installed copy, add its `Scripts` directory to `P py -3.11 -m venv .venv ``` 3. Activate it (`.venv\Scripts\activate.bat`) and confirm `python --version` prints `3.11.x` **and** `where pip` points inside `...\cotdata\.venv\Scripts\`. -4. Reinstall into the venv: `pip install "cotdata[norgate]"`. `cotdata-update` now lands in `.venv\Scripts\` (on PATH while activated). +4. Reinstall into the venv: `pip install -e . && pip install "crucible-marketdata[norgate]"`. `cotdata-update` and `marketdata-update` now land in `.venv\Scripts\` (on PATH while activated). ### Virtual Environment Won't Activate @@ -311,7 +320,7 @@ If you'd rather keep the `pip`-installed copy, add its `Scripts` directory to `P ### The install succeeded but a console script is missing -Symptom: `pip install -e ".[norgate]"` printed success and exited 0, but +Symptom: `pip install -e .` printed success and exited 0, but `.venv\Scripts\cotdata-prices.exe` (or `cotdata-cot.exe`, or `cotdata-update.exe` after a fresh setup) is not there. @@ -334,7 +343,7 @@ If that path is not inside your `.venv\Scripts\`, the install went elsewhere. **Fix:** use uv's own installer, which always targets the active uv venv. ```powershell -uv pip install -e ".[norgate]" +uv pip install -e . ``` Then confirm the executables exist: @@ -353,7 +362,7 @@ disk, this is almost always why. **Problem:** `import cotdata` fails **Likely causes:** -1. **Installed with plain `pip` into a `uv` venv** (most common with uv) → a uv venv has no `pip`, so `pip install` used your *global* pip and installed cotdata outside the venv. Reinstall from inside the clone with `uv pip install -e ".[norgate]"`. Confirm with `where pip` (points at global) vs `where python` (points at `...\.venv\Scripts\`). +1. **Installed with plain `pip` into a `uv` venv** (most common with uv) → a uv venv has no `pip`, so `pip install` used your *global* pip and installed cotdata outside the venv. Reinstall from inside the clone with `uv pip install -e .`. Confirm with `where pip` (points at global) vs `where python` (points at `...\.venv\Scripts\`). 2. Virtual environment not activated (no `(.venv)`/`(cotdata)` in prompt) → run `.venv\Scripts\activate.bat` 3. Installation failed → try `pip install --upgrade pip`, then re-run the [Step 3](#step-3-install-cotdata) install for your role 4. Wrong Python being used → run `python -m pip list` and check cotdata is there @@ -373,18 +382,20 @@ disk, this is almost always why. ### Norgate Errors on Producer Machine -**Problem:** `cotdata-update --prices` fails with "Norgate not found" +**Problem:** `marketdata-update --bars --domain futures` fails with "Norgate not found" **Fix:** -1. You installed with the `[norgate]` extra (`pip install -e ".[norgate]"`)? Check: `pip list | findstr norgatedata` +0. You are running the right command? `cotdata-update --prices` no longer exists — Norgate + bars moved to `marketdata` in ADR-0007. +1. You installed with the `[norgate]` extra (`pip install "crucible-marketdata[norgate]"`)? Check: `pip list | findstr norgatedata` 2. Norgate Data Updater installed and running? Look for "Norgate Data Updater" in the Start menu and open it 3. If Norgate says "authentication required," sign in with your Norgate account 4. Test with: `python -c "from norgatedata import Norgate; print(Norgate)"` ## Next Steps -- **Read data:** See the [README](../README.md#reading-data-consumer) for `get_prices()`, `get_cot()`, and adjustments -- **Produce prices:** If you have Norgate, follow the [Producing data](../README.md#producing-data-producer) section +- **Read data:** See the [README](../README.md#reading-data-consumer) for `get_cot()`, and `marketdata.get_bars()` for bars +- **Produce data:** Follow the [Producing data](../README.md#producing-data-producer) section - **Schedule runs:** Automate daily updates with [Task Scheduler](WINDOWS_SCHEDULING.md) - **Develop locally:** Clone the repo, install with `pip install -e .`, and run tests with `pytest` @@ -399,28 +410,30 @@ python -c "import cotdata; df = cotdata.get_cot('ES'); print(df)" No credentials needed; reads freely from the store. -### Production (Norgate Prices, Windows) +### Production (Norgate Bars, Windows) ```cmd -set COTDATA_STORE=\\shared\cotdata_store -set PYTHONPATH=%PYTHONPATH%;C:\code\cotdata -cotdata-update --prices --metadata --require-final +set MARKETDATA_STORE=\\shared\marketdata_store +marketdata-update --bars --domain futures --require-final +marketdata-update --metadata ``` -Norgate Data Updater must be running. Set `COTDATA_STORE` to a network share if multiple machines read from it. +Norgate Data Updater must be running. Note this is `MARKETDATA_STORE`, a separate directory +from `COTDATA_STORE` — pointing both at one path merges two producers' layouts into one +folder, which is not a supported configuration. Either can be a network share if multiple +machines read from it. ### Server (Databento Prices, Cross-Platform) ```cmd set COTDATA_STORE=C:\data\cotdata_store -set COTDATA_PRICE_SOURCE=databento set DATABENTO_API_KEY=db-... cotdata-update --ingest-databento cotdata-update --build-databento cotdata-update --cot-all ``` -No Norgate needed. Databento is slower but works anywhere. +No Norgate needed. Databento is slower but works anywhere. Read the result back with `cotdata.store.read_prices(symbol, adjustment)`; markets it does not cover (ICE softs, lumber, MSCI proxies) come from `marketdata`. ## Getting Help diff --git a/docs/databento_norgate_parity.md b/docs/databento_norgate_parity.md index c4be096..b67fe25 100644 --- a/docs/databento_norgate_parity.md +++ b/docs/databento_norgate_parity.md @@ -24,12 +24,18 @@ COTDATA_STORE=~/code/cotdata_store_server \ COTDATA_DATABENTO_RAW=~/code/cotdata_store/_raw/databento \ cotdata-update --build-databento -COTDATA_STORE=~/code/cotdata_store python scripts/validate_databento_vs_norgate.py \ - --norgate-store ~/code/cotdata_store \ +python scripts/validate_databento_vs_norgate.py \ + --norgate-store ~/code/marketdata_store \ --databento-store ~/code/cotdata_store_server \ --symbols <...> ``` +> **Since ADR-0007 step 2** the Norgate side of this comparison is a `$MARKETDATA_STORE` +> (`bars/futures/norgate/`), not a cotdata store — the run above was made before the split +> and its `--norgate-store` pointed at `~/code/cotdata_store`. Both harnesses read either +> layout, so the numbers below reproduce against either copy. The databento side is +> unchanged: it still writes `prices/` under `$COTDATA_STORE`. + ## Result: three buckets | Bucket | N | Symbols | Reading | @@ -83,7 +89,7 @@ so it is a cheap one-request-per-rule pull, no statistics), reads the roll dates ``` DATABENTO_API_KEY=... python scripts/investigate_databento_roll_rule.py \ - --norgate-store ~/code/cotdata_store \ + --norgate-store ~/code/marketdata_store \ --symbols CL ZS NG HE ZC LE --tol-days 3 ``` diff --git a/docs/design/finals_ready_data_driven.md b/docs/design/finals_ready_data_driven.md index 8225bcc..0ff979e 100644 --- a/docs/design/finals_ready_data_driven.md +++ b/docs/design/finals_ready_data_driven.md @@ -5,6 +5,17 @@ answered the open question (Norgate never shows an in-progress session's bar); t wired data-driven with unit tests. What remains is a few nights confirming it flips ready at the right moment against the live feed. +**MOVED, 2026-08-09.** The mechanism argued for here now lives in `marketdata` +(`marketdata/providers/norgate.py`, reached by +`marketdata-update --bars --domain futures --require-final`), because ADR-0007 step 2 moved +bar production out of `cotdata` — §7.1 ported the code and §7.5 deleted the copy here. The +design is unchanged and the reasoning below still holds; only the package and the CLI moved. +Two details this document states are now out of date on their face: `--final-cutoff` no +longer exists in either package (§107 kept it as an accepted-but-ignored fallback; §7.5 +removed it with the rest of the flags), and the file path in the Problem section is +`cotdata`'s, which is gone. Left as written, per the doc lifecycle: this is a point-in-time +record of a decision, not a living manual. + ## Problem `--require-final` gates the nightly Norgate price capture so it never stores an diff --git a/docs/design/reading-the-store.md b/docs/design/reading-the-store.md index eb5ba7c..a8b2af1 100644 --- a/docs/design/reading-the-store.md +++ b/docs/design/reading-the-store.md @@ -5,6 +5,12 @@ each of which has already cost a session that reasoned from a plausible-looking plausible-looking count. None of them is about report comparability, which has its own document: [`cross-report-comparability.md`](cross-report-comparability.md). +**Scope note (ADR-0007 step 2).** §1–§3 are about this store. §4 and §5 are about the BAR +store, which moved to [`crucible-marketdata`](https://pypi.org/project/crucible-marketdata/) +along with the providers that fill it. They stay here, updated to the new API, because they +are the reason this document exists: both are naming-and-composition traps that cost real +time, and a reader arriving from a COT question is exactly who walks into them. + --- ## 1. The Disaggregated universe is mostly power and gas basis @@ -81,8 +87,8 @@ precede any differencing, because the other order fails silently. ## 4. Volume: the fuller-sounding parameter is the narrower series -`get_prices` takes `volume="front"` or `volume="reconstructed"`, documented as "continuous -front-month volume" and "true market volume (first + second expiring contract)". +`marketdata.get_bars` takes `volume="front"` or `volume="reconstructed"`, documented as +"continuous front-month volume" and "true market volume (first + second expiring contract)". **The second reads like the fuller series and is the narrower one.** `Volume_Reconstructed = FirstVolume + SecondVolume`, exactly two expiries, while the plain @@ -111,16 +117,21 @@ Three facts that were each written down separately and never composed: 1. **`propadj` is derived on read** from `unadj` + `backadj`. It is not a stored tier. 2. **Norgate is the only vendor supplying all tiers, and it is Windows-only by mechanism rather than by licence**: `norgatedata` talks to a locally installed Norgate Data Updater - application rather than to an API - ([`../../src/cotdata/providers/norgate.py`](../../src/cotdata/providers/norgate.py)). + application rather than to an API (`marketdata/providers/norgate.py`, moved there by + ADR-0007 step 2 §7.1). 3. **databento owes exactly one series per symbol**, `backadj`, because crucible-stack ADR-0007 scopes it to the Linux dashboard's needs and says its coverage "should not be broadened toward parity with Norgate". **Composed: a databento-backed futures store cannot produce `propadj` at all**, so any consumer needing correct percentage returns cannot be served by one. This is a live constraint -on ADR-0007 step 2 rather than a historical note, and it is a **tier fact, not an operating -system fact**, a distinction that has been got wrong more than once: +rather than a historical note, and it is a **tier fact, not an operating system fact**, a +distinction that has been got wrong more than once: + +Step 2 §7.5 sharpened it into a structural one. `marketdata` now refuses to derive `propadj` +unless BOTH stored tiers are present, rather than returning an empty frame — the store cannot +hand back a silently wrong series. databento, which still writes into `$COTDATA_STORE`, does +write both tiers, but there is no bar reader here that derives the third from them. | box | Python >= 3.10 | Norgate | can produce a store carrying all tiers | |---|---|---|---| diff --git a/docs/examples/linux/run-prices.sh b/docs/examples/linux/run-prices.sh index 34b9b09..f5ce770 100755 --- a/docs/examples/linux/run-prices.sh +++ b/docs/examples/linux/run-prices.sh @@ -7,12 +7,14 @@ # (Plain-text markers, not angle-bracket placeholders: an unedited <...> would be # read as a shell redirection. Stage 1 is the paid pull, Stage 2 a free rebuild.) # cotdata-prices runs the PRICE half only and refuses --cot-all. +# Databento is the only price producer left here: ADR-0007 moved the Norgate and +# Yahoo bar producers to `marketdata`, so the markets databento does not cover +# (ICE softs, lumber, MSCI proxies) come from `marketdata-update --bars` against +# $MARKETDATA_STORE, not from this script. # See docs/LINUX_SCHEDULING.md for the crontab and flock setup. set -euo pipefail export COTDATA_STORE=REPLACE_WITH_STORE_PATH -export COTDATA_PRICE_SOURCE=databento export DATABENTO_API_KEY=REPLACE_WITH_DATABENTO_KEY BIN=REPLACE_WITH_VENV_PATH/bin/cotdata-prices "$BIN" --ingest-databento # Stage 1 (paid): raw .n.0/.n.1 to raw store "$BIN" --build-databento # Stage 2 (free): back-adjusted prices -"$BIN" --prices-yahoo # softs / lumber / MSCI fallback diff --git a/docs/examples/mac/pull-store.sh b/docs/examples/mac/pull-store.sh index 96b01a0..a32a71f 100755 --- a/docs/examples/mac/pull-store.sh +++ b/docs/examples/mac/pull-store.sh @@ -34,7 +34,7 @@ EXCLUDES=( ) # Two passes so a manifest never arrives before the data it describes. Harmless if -# reversed (get_prices reads parquet directly), but free to get right. +# reversed (readers open parquet directly), but free to get right. rsync -az --delete "${EXCLUDES[@]}" --exclude 'manifests/' "$SRC/" "$DEST/" rsync -az "$SRC/manifests/" "$DEST/manifests/" diff --git a/docs/examples/windows/push-to-server.cmd b/docs/examples/windows/push-to-server.cmd index 8e035b0..253e250 100644 --- a/docs/examples/windows/push-to-server.cmd +++ b/docs/examples/windows/push-to-server.cmd @@ -48,7 +48,7 @@ set "DEST=REPLACE_WITH_REMOTE" set "SSH=%SSH_EXE% -i %KEY% -o BatchMode=yes -o StrictHostKeyChecking=accept-new -o UserKnownHostsFile=%KNOWN%" REM Data first, manifests last, so a manifest never announces parquet that has not -REM landed (harmless if reversed; get_prices reads parquet directly). --delete makes +REM landed (harmless if reversed; readers open parquet directly). --delete makes REM this a true mirror. The exclusions match the Mac push: REM _cache, _raw producer-internal; _raw/databento (the paid databento bronze) REM rides under _raw and so is excluded, per ADR-0006. diff --git a/docs/examples/windows/run-prices.cmd b/docs/examples/windows/run-prices.cmd index e71fc36..14cac4a 100644 --- a/docs/examples/windows/run-prices.cmd +++ b/docs/examples/windows/run-prices.cmd @@ -1,14 +1,20 @@ @echo off -REM cotdata price update wrapper for Windows Task Scheduler. +REM marketdata bar update wrapper for Windows Task Scheduler. REM Copy this file into your scheduler folder and overwrite the two markers below. REM Do NOT put angle brackets in a .cmd file: cmd reads them as redirection and REM the file fails with "The syntax of the command is incorrect" even on comment REM lines, which is why these are plain-text markers you replace. -REM REPLACE_WITH_STORE_PATH = your data store e.g. C:\Users\you\cotdata_store -REM REPLACE_WITH_VENV_PATH = your cotdata venv e.g. C:\Users\you\code\cotdata\.venv -REM cotdata-prices runs the PRICE half only and refuses --cot-all, so this box +REM REPLACE_WITH_MARKETDATA_STORE_PATH = your BAR store e.g. C:\Users\you\marketdata_store +REM REPLACE_WITH_VENV_PATH = your venv e.g. C:\Users\you\code\cotdata\.venv +REM +REM ADR-0007 moved bar production out of cotdata: this now runs marketdata-update +REM against MARKETDATA_STORE, which is a DIFFERENT directory from COTDATA_STORE, +REM not an alias for it. The COT half still runs from run-cot.cmd, so this box REM cannot become a second COT producer racing whatever already does that job. -REM --require-final defers (exits non-zero) until Norgate's Final prices are in; -REM pair it with Task Scheduler "restart on failure". See docs/WINDOWS_SCHEDULING.md. -set COTDATA_STORE=REPLACE_WITH_STORE_PATH -"REPLACE_WITH_VENV_PATH\Scripts\cotdata-prices.exe" --prices --metadata --require-final +REM +REM --require-final defers (exits non-zero, having fetched nothing) until Norgate +REM holds a newer settled bar than the store does; pair it with Task Scheduler +REM "restart on failure". See docs/WINDOWS_SCHEDULING.md. +set MARKETDATA_STORE=REPLACE_WITH_MARKETDATA_STORE_PATH +"REPLACE_WITH_VENV_PATH\Scripts\marketdata-update.exe" --bars --domain futures --require-final +"REPLACE_WITH_VENV_PATH\Scripts\marketdata-update.exe" --metadata diff --git a/docs/handoffs/2026-08-04-adr7-step2-price-producer-split.md b/docs/handoffs/2026-08-04-adr7-step2-price-producer-split.md index 2959ed1..8735574 100644 --- a/docs/handoffs/2026-08-04-adr7-step2-price-producer-split.md +++ b/docs/handoffs/2026-08-04-adr7-step2-price-producer-split.md @@ -209,9 +209,10 @@ Step 3 stays out. `npf` and `livebook` are a separate pass with a live book behi ## 8. Outcome, appended 2026-08-08 -**Step 1 of §7 executed** (`marketdata` PR #7), and §7.2 came with it. §7.3–§7.5 are not -started. Full record: -[`2026-08-08-adr7-step2-provider-shipped.md`](2026-08-08-adr7-step2-provider-shipped.md). +**§7 is complete.** Step 1 shipped as `marketdata` PR #7 and §7.2 came with it (full record: +[`2026-08-08-adr7-step2-provider-shipped.md`](2026-08-08-adr7-step2-provider-shipped.md)); +§7.3 was voided by §8.2; §7.4 and §7.5 landed 2026-08-09. §8.5 records what §7.5 actually +deleted and the one thing it deliberately did not. Body preserved verbatim above, per the register convention. This section carries the corrections; §4 and §7.3 carry pointers to it. @@ -303,3 +304,56 @@ reporting `missing: specs,unadj_price,backadj_price`, pinned by `tests/test_contract_master_live.py` as the only two non-joinable, absent from both vintage panels per the 2026-08-04 spec inventory, and `Role: heldout` in the deployed `params.yaml`. The counts agree: 49 of 51 joinable, and 49 futures symbols ported. + +### 8.5 What §7.5 deleted, and the exception it kept + +Landed 2026-08-09. Gone from `cotdata`: `prices.py` (including the derived `propadj` tier), +`providers/norgate.py`, `providers/yfinance.py`, the `get_prices` / `roll_dates` exports, +`store.{write,upsert,read}_metadata`, the `--prices` / `--metadata` / `--prices-yahoo` / +`--require-final` / `--final-cutoff` / `--full` CLI flags, the `norgate` and `yahoo` extras, +and the `packaging` runtime dependency that existed only for `norgatedata`. + +**`providers/databento.py` stays**, and with it `store.write_prices` / `read_prices`, +`config.prices_dir()` and the `prices` manifest half it writes through. §7.5 was scoped to +"delete the price surface", and databento is inside that surface by shape but outside it by +argument: ADR-0006 accepted it as a *validated provider-different alternative*, it is the +fleet's only intraday-capable source, and ADR-0007 never scoped a marketdata equivalent. So +`cotdata-prices` survives, scoped to `--ingest-databento` / `--build-databento`. The +*consumer* bar API left; the store-level pair the retained producer writes through did not. +That is the line, and it is worth stating because "cotdata is COT-only" is now true of its +public API and not yet of its store. + +**Three findings the plan did not anticipate**, each a case of the deletion exposing +something §7.1 had left: + +1. **`finals_ready()` had no caller.** It was ported in §7.1 but never wired to a CLI flag, + so `cotdata-update --prices --require-final` was the only way to reach it. Deleting that + would have left the Windows nightly job ungated, and the failure it prevents is silent — + a fetch before Norgate settles writes a provisional bar over a real one with nothing in + the store to say so. Fixed first, as `marketdata` PR #13, because the deletion was not + safe without it. + +2. **Six behaviours had tests only in `cotdata`.** §7.1 ported the provider and not its + tests; `marketdata`'s suite covered the pure functions and left everything reachable only + through `update()` untested — volume reconstruction, the volume-rank pick, the + incremental window, `full=True`, the NDU-down abort, the all-null spec-row skip. Deleting + here would have been the moment those stopped being tested anywhere. Ported in the same + PR. **A file-count check would have missed this entirely**: the test *files* existed on + both sides, with zero name overlap and a real coverage gap underneath. + +3. **Two harnesses read the Norgate store by path.** + `scripts/validate_databento_vs_norgate.py` and + `scripts/investigate_databento_roll_rule.py` open `prices/_backadj.parquet` + directly rather than through an API, so no call-site grep finds them — the same class of + coupling as §8.3, one layer out. Both now read `bars/futures/norgate/` first and fall + back to the old layout, so the ADR-0006 parity gate still runs across the split. + +**Known breakage, accepted.** `crowdmon` (frozen, archived) and +`npf/docs/crowdmon/reproduce_forced_flow_mechanism.py` still call `cotdata.get_prices`. +Both are point-in-time records under their repos' doc lifecycle and were left untouched on +purpose; §8.2 already voided the `crowdmon` repoint on the same grounds. + +**Still open for `crucible-stack`:** ADR-0007's "Status of the work" needs updating, and it +now has a second decision to record — that databento remains in `cotdata` with no +marketdata equivalent, which is a live exception to the ADR's own boundary rather than a +step still to do. diff --git a/pyproject.toml b/pyproject.toml index 084c7b2..055b1fe 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,8 +4,8 @@ build-backend = "setuptools.build_meta" [project] name = "cotdata" -version = "0.3.0" -description = "Canonical data layer for COT/futures strategies — a producer/consumer split over a file-based store." +version = "0.4.0" +description = "Canonical CFTC positioning (COT) data layer — a producer/consumer split over a file-based store." readme = "README.md" authors = [ {name = "Matt S.", email = "cotanalyzer@gmail.com"} @@ -21,7 +21,6 @@ dependencies = [ "pyarrow>=10", "requests>=2.28", # CFTC COT zips "python-dateutil>=2.8", - "packaging", # required by norgatedata "xlrd>=2.0", # CFTC dea_fut_xls are legacy .xls "PyYAML>=6.0", ] @@ -32,13 +31,12 @@ Repository = "https://github.com/mspinola/cotdata" Issues = "https://github.com/mspinola/cotdata/issues" [project.optional-dependencies] -# Producer-only, Windows: the Norgate Data Updater must be installed & running. -norgate = ["norgatedata"] # Cross-platform price producer: two-stage ingest/build back-adjustment (see the # Providers section of the README and ADR-0006). Also serves the intraday work. +# The `norgate` and `yahoo` extras moved to crucible-marketdata with the bar +# producers they installed for (ADR-0007). databento = ["databento>=0.30"] -yahoo = ["yfinance"] dev = ["pytest>=7", "ruff==0.15.22"] [project.scripts] diff --git a/requirements.txt b/requirements.txt index 51914e8..2cf1db1 100644 --- a/requirements.txt +++ b/requirements.txt @@ -5,10 +5,10 @@ pandas>=1.5 pyarrow>=10 requests>=2.28 python-dateutil>=2.8 -packaging xlrd>=2.0 -# Windows producer (Norgate) additionally needs the optional extra: -# uv pip install -e ".[norgate]" (or: pip install norgatedata) -# Dormant intraday/settlement-crosscheck provider: +# Bars are NOT here: the Norgate and Yahoo producers moved to crucible-marketdata +# (ADR-0007), and so did their `norgate` / `yahoo` extras. +# The databento price provider stays (ADR-0006 alternative producer + the intraday +# work) and needs its optional extra: # uv pip install -e ".[databento]" diff --git a/scripts/investigate_databento_roll_rule.py b/scripts/investigate_databento_roll_rule.py index 37048ca..30ed2bb 100644 --- a/scripts/investigate_databento_roll_rule.py +++ b/scripts/investigate_databento_roll_rule.py @@ -26,9 +26,12 @@ re-run, or extra --tol-days values, cost no extra pulls. Pull a symbol once, then sweep tolerances for free. +Since ADR-0007 the Norgate store is a $MARKETDATA_STORE (`bars/futures/norgate/`); +the pre-move cotdata `prices/` layout is still read, so an older synced copy works. + Usage: DATABENTO_API_KEY=... python scripts/investigate_databento_roll_rule.py \ - --norgate-store ~/code/cotdata_store \ + --norgate-store ~/code/marketdata_store \ --symbols CL ZS NG HE --tol-days 3 7 10 # sweep, one pull per rule """ from __future__ import annotations @@ -50,8 +53,11 @@ def norgate_roll_dates(store: str, symbol: str) -> pd.DatetimeIndex: """Roll dates from a Norgate-built store: the first session on each new front contract, read from the `Delivery Month` column of the backadj parquet.""" - p = Path(store) / "prices" / f"{symbol}_backadj.parquet" - if not p.exists(): + for layout in ("bars/futures/norgate", "prices"): # marketdata, then pre-ADR-0007 + p = Path(store) / layout / f"{symbol}_backadj.parquet" + if p.exists(): + break + else: return pd.DatetimeIndex([]) df = pd.read_parquet(p) if "Delivery Month" not in df.columns: @@ -117,7 +123,8 @@ def score(dbrolls: pd.DatetimeIndex, ngrolls: pd.DatetimeIndex, tol_days: int): def main() -> None: ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) - ap.add_argument("--norgate-store", required=True, help="A Norgate-built cotdata store.") + ap.add_argument("--norgate-store", required=True, + help="A Norgate-built store — a $MARKETDATA_STORE since ADR-0007.") ap.add_argument("--symbols", nargs="+", default=_DEFAULT_SYMBOLS) ap.add_argument("--rules", nargs="+", default=_DEFAULT_RULES, help="c=calendar n=OI v=volume") ap.add_argument("--tol-days", nargs="+", type=int, default=[3], diff --git a/scripts/probe_norgate_finals.py b/scripts/probe_norgate_finals.py deleted file mode 100644 index 37d995f..0000000 --- a/scripts/probe_norgate_finals.py +++ /dev/null @@ -1,68 +0,0 @@ -#!/usr/bin/env python -"""Probe norgatedata's data-content signals for the data-driven finals_ready redesign. - -Run on the WINDOWS producer (needs norgatedata + a live Norgate subscription). - -Run it TWICE and compare the output: - 1. during the trading day (before Norgate's evening publish), and - 2. after the evening publish (e.g. ~9:30 PM ET). - -The key question: does `last_quoted_date` for a continuous symbol show TODAY's date -only after settlement, or already during the day (a provisional bar)? That decides -whether bar-presence alone is a sufficient "finals are in" signal. See -docs/design/finals_ready_data_driven.md. -""" -import datetime as dt - -import norgatedata - -REF_SYMBOLS = ["&ES", "&CL", "&ZC"] # liquid continuous: S&P, WTI, corn — trade every session -FINAL_DATABASES = ["Futures", "Continuous Futures"] - -print(f"# probe at {dt.datetime.now().isoformat()} (local PC time)") -print(f"norgatedata version: {getattr(norgatedata, '__version__', '?')}") -try: - print(f"NDU running (status()): {norgatedata.status()}") -except Exception as e: # noqa: BLE001 - print(f"status() ERROR: {e!r}") - -# 1) Confirm whether this build has ANY calendar/holiday/session function. -api = sorted(x for x in dir(norgatedata) if not x.startswith("_")) -cal = [x for x in api if any(k in x.lower() for k in - ("holiday", "calendar", "session", "business", "market_day", "trading_day", "busday"))] -print(f"\ncalendar-ish functions in this build: {cal or 'NONE'}") -print(f"full API: {api}") - -# 2) Database refresh times — the current (fragile) signal. -print("\n-- last_database_update_time --") -for db in FINAL_DATABASES: - try: - print(f" {db}: {norgatedata.last_database_update_time(db)}") - except Exception as e: # noqa: BLE001 - print(f" {db}: ERROR {e!r}") - -# 3) Data-content signals per ref symbol — the proposed signal. -print("\n-- per-symbol date/time signals --") -for s in REF_SYMBOLS: - try: - lq = norgatedata.last_quoted_date(s, datetimeformat="iso") - slq = norgatedata.second_last_quoted_date(s, datetimeformat="iso") - lpu = norgatedata.last_price_update_time(s) - print(f" {s}: last_quoted_date={lq} second_last={slq} last_price_update_time={lpu}") - except Exception as e: # noqa: BLE001 - print(f" {s}: ERROR {e!r}") - -# 4) The actual tail — does today's bar exist yet, and do its values look settled? -print("\n-- price_timeseries tail (last 4 bars) --") -for s in REF_SYMBOLS: - try: - df = norgatedata.price_timeseries( - s, - padding_setting=norgatedata.PaddingType.NONE, - timeseriesformat="pandas-dataframe", - start_date="2026-07-20", - ) - print(f" {s}:") - print(" " + df.tail(4).to_string().replace("\n", "\n ")) - except Exception as e: # noqa: BLE001 - print(f" {s}: ERROR {e!r}") diff --git a/scripts/reconcile_volume.py b/scripts/reconcile_volume.py deleted file mode 100644 index 0fba12b..0000000 --- a/scripts/reconcile_volume.py +++ /dev/null @@ -1,117 +0,0 @@ -import argparse - -import numpy as np -import pandas as pd - -from cotdata import store -from cotdata.providers import norgate - - -def get_roll_windows(df: pd.DataFrame, window_days: int = 5) -> pd.DataFrame: - """Find dates where the contract rolled (Delivery Month changed) and return a mask of those dates +/- window_days.""" - if "Delivery Month" not in df.columns: - return pd.Series(False, index=df.index) - - dm = df["Delivery Month"] - roll = dm.ne(dm.shift()) & dm.shift().notna() - - # Expand window - roll_dates = df.index[roll] - mask = pd.Series(False, index=df.index) - for rd in roll_dates: - start = rd - pd.Timedelta(days=window_days) - end = rd + pd.Timedelta(days=window_days) - mask.loc[start:end] = True - - return mask - -def verify_symbol(symbol: str): - print(f"\n{'='*50}\nReconciling {symbol}\n{'='*50}") - - print(f"1. Fetching & Reconstructing {symbol} via norgate.py ...") - norgate.update(symbols=[symbol]) - - print(f"2. Reading updated dataframe for {symbol} ...") - df = store.read_prices(symbol, "backadj") - - if df.empty: - print(f"Error: No data found for {symbol}.") - return - - # Check Columns - expected_cols = ["Volume", "Volume_Reconstructed", "FirstVolume", "SecondVolume", "Volume_Source", "FirstContract", "SecondContract"] - for col in expected_cols: - if col not in df.columns: - print(f"❌ Missing expected column: {col}") - return - - print("✅ All expected additive columns are present.") - - # Check Fallback - source = df["Volume_Source"].iloc[-1] - print(f"Volume Source: {source}") - - if source == "raw": - print(f"✅ Fallback applied correctly. Volume_Reconstructed == Volume for all rows: {(df['Volume_Reconstructed'] == df['Volume']).all()}") - return - - # Check bounds - print("3. Running Sanity Checks...") - - # a. First + Second = Reconstructed - valid = df.dropna(subset=["FirstVolume", "SecondVolume"]) - reconstructed_match = np.isclose( - valid["FirstVolume"] + valid["SecondVolume"], - valid["Volume_Reconstructed"] - ).all() - print(f"{'✅' if reconstructed_match else '❌'} FirstVolume + SecondVolume == Volume_Reconstructed") - - # b. Reconstructed >= Volume (raw) - # The true combined volume should be roughly >= the single front-month volume - # Some minor exceptions can happen due to Norgate raw volume inclusions, but generally true. - vol_diff = (valid["Volume_Reconstructed"] >= valid["Volume"] * 0.95).mean() - print(f"{'✅' if vol_diff > 0.99 else '❌'} Volume_Reconstructed is >= Raw Volume (in {vol_diff*100:.1f}% of days)") - - # c. No NaNs introduced into default Volume - nans_in_raw = df["Volume"].isna().sum() - print(f"{'✅' if nans_in_raw == 0 else '❌'} Default 'Volume' has 0 NaNs ({nans_in_raw} found)") - - print("\n4. Roll Window Drop-off Analysis") - # Compare average volume during roll windows vs non-roll windows - roll_mask = get_roll_windows(df, window_days=5) - - if roll_mask.sum() == 0: - print("No roll windows detected.") - return - - roll_raw = df.loc[roll_mask, "Volume"].mean() - nonroll_raw = df.loc[~roll_mask, "Volume"].mean() - - roll_rec = df.loc[roll_mask, "Volume_Reconstructed"].mean() - nonroll_rec = df.loc[~roll_mask, "Volume_Reconstructed"].mean() - - raw_drop = (nonroll_raw - roll_raw) / nonroll_raw * 100 if nonroll_raw else 0 - rec_drop = (nonroll_rec - roll_rec) / nonroll_rec * 100 if nonroll_rec else 0 - - print(" Raw Volume:") - print(f" Non-Roll Avg: {nonroll_raw:,.0f}") - print(f" Roll Avg: {roll_raw:,.0f}") - print(f" Drop-off: {raw_drop:.1f}%") - - print(" Reconstructed Volume:") - print(f" Non-Roll Avg: {nonroll_rec:,.0f}") - print(f" Roll Avg: {roll_rec:,.0f}") - print(f" Drop-off: {rec_drop:.1f}%") - - if rec_drop < raw_drop: - print("✅ Reconstructed volume successfully smoothed the roll drop-off!") - else: - print("❌ Reconstructed volume did not improve the roll drop-off.") - -if __name__ == "__main__": - parser = argparse.ArgumentParser(description="Reconcile Volume Reconstruction") - parser.add_argument("--symbols", nargs="+", default=["ES", "BTC"], help="Internal symbols to test") - args = parser.parse_args() - - for sym in args.symbols: - verify_symbol(sym) diff --git a/scripts/validate_databento_vs_norgate.py b/scripts/validate_databento_vs_norgate.py index 85f5871..23b5c17 100644 --- a/scripts/validate_databento_vs_norgate.py +++ b/scripts/validate_databento_vs_norgate.py @@ -24,10 +24,15 @@ It needs real data, so it is NOT run in CI — the comparison logic is unit tested in tests/test_validate_databento.py against synthetic frames. +Since ADR-0007 the two stores belong to two packages: the Norgate side is a +$MARKETDATA_STORE (`bars/futures/norgate/`), the databento side is a $COTDATA_STORE +(`prices/`). `read_backadj` accepts either layout, so a Norgate store synced before +the move still reads. + Usage: python scripts/validate_databento_vs_norgate.py \ - --norgate-store /path/to/norgate_store \ - --databento-store /path/to/databento_store \ + --norgate-store /path/to/marketdata_store \ + --databento-store /path/to/cotdata_store \ --symbols ES CL GC """ from __future__ import annotations @@ -48,16 +53,26 @@ _DEFAULT_MAX_OUTLIER_DAYS = 8 # days/yr allowed to exceed REL_TOL (roll-date mismatches) +# Where a store keeps `_backadj.parquet`, newest layout first. +# ADR-0007 moved the Norgate side into `marketdata`, which keys a series by +# `bars///`; the databento side still writes cotdata's flat +# `prices/`. Both are read here rather than one, because this harness compares two +# stores that are now produced by two different packages — and because a Norgate +# store synced before the move still has the old shape. +_LAYOUTS = ("bars/futures/norgate", "prices") + + def read_backadj(store_path: str, symbol: str) -> pd.DataFrame: - """Read one symbol's backadj OHLC frame straight from a store's parquet, normalized - the way get_prices would (tz-naive daily Date index). Empty if absent.""" - p = Path(store_path) / "prices" / f"{symbol}_backadj.parquet" - if not p.exists(): - return pd.DataFrame() - df = pd.read_parquet(p) - df.index = pd.to_datetime(df.index).tz_localize(None).normalize() - df.index.name = "Date" - return df.sort_index() + """Read one symbol's backadj OHLC frame straight from a store's parquet, + normalized to a tz-naive daily Date index. Empty if absent.""" + for layout in _LAYOUTS: + p = Path(store_path) / layout / f"{symbol}_backadj.parquet" + if p.exists(): + df = pd.read_parquet(p) + df.index = pd.to_datetime(df.index).tz_localize(None).normalize() + df.index.name = "Date" + return df.sort_index() + return pd.DataFrame() def _roll_dates(df: pd.DataFrame) -> pd.DatetimeIndex: @@ -152,7 +167,9 @@ def format_report(m: dict, fails: list) -> str: def main(argv=None) -> int: p = argparse.ArgumentParser(description="Validate databento backadj against Norgate.") - p.add_argument("--norgate-store", required=True, help="Path to the Norgate-built store.") + p.add_argument("--norgate-store", required=True, + help="Path to the Norgate-built store — a $MARKETDATA_STORE since " + "ADR-0007. The pre-move cotdata layout is still accepted.") p.add_argument("--databento-store", required=True, help="Path to the databento-built store.") p.add_argument("--symbols", nargs="+", default=_DEFAULT_SYMBOLS) p.add_argument("--change-corr-min", type=float, default=_DEFAULT_CHANGE_CORR_MIN) diff --git a/src/cotdata/__init__.py b/src/cotdata/__init__.py index f738239..1838fa0 100644 --- a/src/cotdata/__init__.py +++ b/src/cotdata/__init__.py @@ -1,12 +1,17 @@ -"""cotdata — canonical data layer (see README). Public consumer API.""" +"""cotdata — canonical CFTC positioning layer (see README). Public consumer API. + +Bars are NOT here. ADR-0007 makes this package COT-only and moves every price +series to `marketdata`, so `get_prices`/`roll_dates` are gone — import +``marketdata.get_bars`` instead. The two packages keep separate stores +($COTDATA_STORE, $MARKETDATA_STORE) and separate producers. +""" from .cot import get_cot -from .prices import get_prices, roll_dates from .registry import REGISTRY, Symbol, all_symbols, symbol from .store import load_manifest, require_schema, schema_version -__version__ = "0.3.0" +__version__ = "0.4.0" __all__ = [ - "get_prices", "roll_dates", "get_cot", + "get_cot", "symbol", "all_symbols", "REGISTRY", "Symbol", "load_manifest", "schema_version", "require_schema", ] diff --git a/src/cotdata/config.py b/src/cotdata/config.py index 742f1df..6d3f1ed 100644 --- a/src/cotdata/config.py +++ b/src/cotdata/config.py @@ -2,11 +2,12 @@ import os from pathlib import Path -# v2 — reconstructed volume promoted: prices carry Volume_Reconstructed / -# Volume_Source, and get_prices(volume="reconstructed") serves them. The store -# was migrated by a full producer pass (2026-07-14) and now carries v2 shape; -# schema_version() reflects the on-disk manifest, so a fresh/partial store can -# still read <2. +# v2 — reconstructed volume promoted: bars carry Volume_Reconstructed / +# Volume_Source. The reader that serves them moved to marketdata with the bars +# themselves (ADR-0007); this number stays because it stamps THIS store, and a +# consumer keys cache invalidation on it. The store was migrated by a full +# producer pass (2026-07-14) and now carries v2 shape; schema_version() reflects +# the on-disk manifest, so a fresh/partial store can still read <2. SCHEMA_VERSION = 2 diff --git a/src/cotdata/prices.py b/src/cotdata/prices.py deleted file mode 100644 index dfb383a..0000000 --- a/src/cotdata/prices.py +++ /dev/null @@ -1,186 +0,0 @@ -"""Consumer price API. Reads the store and returns the shape downstream code -expects (the old fetch_daily_ohlc contract). No network, cross-platform.""" -from typing import Optional - -import numpy as np -import pandas as pd - -from . import store - -_COLS = ["Open", "High", "Low", "Close", "Volume", "Open Interest"] -_OHLC = ["Open", "High", "Low", "Close"] - - -def _ratio_adjust(symbol: str) -> pd.DataFrame: - """Derive a proportional (ratio) back-adjusted OHLC series from the stored - unadjusted + additive-back-adjusted series. Empty if either is missing. - - WHY. Norgate only publishes ADDITIVE (arithmetic) back-adjustment — the - ``_CCB`` continuous (see providers/norgate.py). For a low-priced, long-history - contract like DC (Class III Milk, ~$15–20/cwt over ~29y) the additive - accumulation of roll gaps drives ~47% of back-adjusted closes ≤ 0 (down to - −9.83). A close-based stop, an R-multiple, or a % return is meaningless on a - non-positive series, so CMR cannot use DC's ``backadj`` at all. A ratio-adjusted - series preserves percentage returns. - - **It does NOT "stay strictly positive", which this docstring used to claim.** - Ratio adjustment scales each segment by a POSITIVE factor, so it preserves the - sign of the unadjusted series rather than imposing one: where the underlying - market traded below zero, so does the output. WTI settled at −37.63 on - 2020-04-20 and CL's ``propadj`` close that day is −24.11. - - That is one bar in the whole store. Measured across all 47 symbols, ``propadj`` - has exactly ONE non-positive close anywhere (0.009% of crude's 10,882 bars), - against ``backadj``'s 52.3% for ZS and 41.2% for DC. So what this function - actually delivers is **three orders of magnitude fewer non-positive closes**, - not none, and a consumer computing returns has to handle the sign change on that - one day rather than assume it away. Found by a downstream volatility module that - raised on any non-positive close and so refused to produce a number for crude at - all, across its entire 43-year history, over one real trading day. - - This is a consumer-side derivation (no network, no Windows/norgatedata), in the - same spirit as the reconstructed-volume view: the transform lives in the data - layer so consumers ask for what they want. It is a pure function of two series - already in the store, so it needs no producer re-run or schema bump. - - METHOD. The additive series ``B`` and the unadjusted series ``U`` differ by an - offset ``O = B − U`` that is piecewise-constant and steps only at rolls (each - roll's step is Norgate's stitched calendar spread; verified on DC — every step - >$0.0001 lands on a Delivery-Month change). At roll ``r`` the recovered spread, - measured on the last day the OLD contract is front (day ``r−1``), is - ``s = O[r−1] − O[r] = F_new(r−1) − F_old(r−1)``; the roll ratio is - ``k = (U[r−1] + s) / U[r−1] = F_new/F_old``. Each historical segment is scaled - by the cumulative product of ``k`` for all rolls at/after it, anchoring the - most-recent segment to 1 (actual prices). Result: identical daily % returns to - ``U`` within a segment, gap-free true returns across each roll (sign-identical - to ``B``), strictly positive, anchored to the real current price. O/H/L/C are - scaled by the per-row segment factor; Volume/Open Interest/Delivery Month and - the reconstruction columns pass through from the unadjusted frame unchanged. - """ - U = store.read_prices(symbol, "unadj") - B = store.read_prices(symbol, "backadj") - if U.empty or B.empty or "Close" not in U or "Close" not in B: - return pd.DataFrame() - - U = U.copy() - U.index = pd.to_datetime(U.index).tz_localize(None).normalize() - B.index = pd.to_datetime(B.index).tz_localize(None).normalize() - U = U.sort_index() - idx = U.index.intersection(B.index.sort_values()) - if len(idx) == 0: - return pd.DataFrame() - U = U.loc[idx] - b_close = B["Close"].reindex(idx) - - # Roll boundaries: prefer the semantic Delivery-Month change (matches - # roll_dates); fall back to material offset steps when it is absent. - offset = b_close - U["Close"] - if "Delivery Month" in U.columns: - dm = U["Delivery Month"] - roll = dm.ne(dm.shift()) & dm.shift().notna() - else: - roll = offset.diff().abs() > 1e-4 - roll.iloc[0] = False - - seg = roll.cumsum() # segment id, increments at each roll - spread = (-offset.diff()).where(roll, 0.0) # F_new(r−1) − F_old(r−1) - u_prev = U["Close"].shift(1) - ratio = pd.Series(1.0, index=idx) - ok = roll & (u_prev > 0) - ratio[ok] = (u_prev[ok] + spread[ok]) / u_prev[ok] - - # Per-segment cumulative factor, anchored so the most recent segment = 1 - # (kept at actual prices). Walk rolls back-to-front, compounding each ratio. - roll_ratios = ratio[roll].to_numpy() - n_seg = int(seg.iloc[-1]) - factors = np.ones(n_seg + 1) - for s in range(n_seg - 1, -1, -1): - factors[s] = factors[s + 1] * roll_ratios[s] - factor = pd.Series(factors[seg.to_numpy()], index=idx) - - out = U.copy() - for c in _OHLC: - if c in out.columns: - out[c] = out[c] * factor - return out - - -def get_prices(symbol: str, adjustment: str = "backadj", - start: Optional[str] = None, - volume: str = "front") -> pd.DataFrame: - """Daily bars for `symbol`. - - adjustment: - 'backadj' : additive (arithmetic) back-adjustment — settlement close, - gap-free rolls, shape-preserving. Preserves absolute daily - price *changes*. Default for signals + stops. - 'unadj' : raw front-month prices (absolute price / point-value sizing). - 'propadj' : proportional (ratio) back-adjustment, DERIVED on read from - unadj + backadj (see `_ratio_adjust`). Preserves daily *percent* - returns. The series to use for VOLATILITY, and for low-priced, - long-history contracts (e.g. DC / Class III Milk) where additive - back-adjustment accumulates roll gaps below zero and breaks - price-based stops and R-multiples. Note it is not *strictly* - positive (this line used to say it was): ratio adjustment scales - by a positive factor, so it preserves the underlying sign, and - CL 2020-04-20 is −24.11 because WTI really settled at −37.63. - One bar in the whole store; see `_ratio_adjust`. - - volume: which series the `Volume` column carries — - 'front' : continuous front-month volume as Norgate reports it - (default — output is byte-identical to the pre-v2 API). - 'reconstructed' : true market volume (first + second expiring contract, - with per-row fall-back to front-month where individual - contracts are unavailable). The intent view: the fall-back - policy lives here, in the data layer, so consumers ask for - what they want instead of re-deriving it. Adds a - 'Volume_Source' column ('reconstructed' / 'raw') for audit. - - Returns Open/High/Low/Close/Volume/Open Interest indexed by tz-naive Date - (plus 'Delivery Month' if the producer carried it, plus 'Volume_Source' when - volume='reconstructed'), or empty if absent. - """ - if volume not in ("front", "reconstructed"): - raise ValueError(f"volume must be 'front' or 'reconstructed', got {volume!r}") - - df = _ratio_adjust(symbol) if adjustment == "propadj" else store.read_prices(symbol, adjustment) - if df.empty: - return df - df.index = pd.to_datetime(df.index).tz_localize(None).normalize() - df.index.name = "Date" - df = df.sort_index() - if start: - df = df[df.index >= pd.Timestamp(start)] - for c in _COLS: - if c not in df.columns: - df[c] = float("nan") - - keep = list(_COLS) - if volume == "reconstructed": - # Swap in reconstructed volume with a per-row raw fall-back. The producer - # already writes Volume_Reconstructed==Volume on 'raw' rows, so reading it - # is fall-back-safe where the column exists; guard for older/partial data - # (pre-reconstruction parquet, or a stray NaN) by filling from front-month. - if "Volume_Reconstructed" in df.columns: - df["Volume"] = df["Volume_Reconstructed"].where( - df["Volume_Reconstructed"].notna(), df["Volume"]) - df["Volume_Source"] = ( - df["Volume_Source"] if "Volume_Source" in df.columns else "reconstructed") - else: - # Store predates reconstruction → everything is front-month. - df["Volume_Source"] = "raw" - keep = keep + ["Volume_Source"] - - if "Delivery Month" in df.columns: - keep = keep + ["Delivery Month"] - return df[keep].copy() - - -def roll_dates(symbol: str, adjustment: str = "backadj") -> pd.DatetimeIndex: - """Exact roll dates = days where the underlying Delivery Month changes. - Empty if the producer didn't carry 'Delivery Month'.""" - df = get_prices(symbol, adjustment) - if df.empty or "Delivery Month" not in df.columns: - return pd.DatetimeIndex([]) - changed = df["Delivery Month"].ne(df["Delivery Month"].shift()) - return df.index[changed.fillna(False)] diff --git a/src/cotdata/providers/norgate.py b/src/cotdata/providers/norgate.py deleted file mode 100644 index 79c59b8..0000000 --- a/src/cotdata/providers/norgate.py +++ /dev/null @@ -1,546 +0,0 @@ -"""Norgate price producer — RUNS ON WINDOWS (Norgate Data Updater running + -`norgatedata`). The active EOD source: exchange settlement close, deep history, -back-adjusted (gap-free, shape-preserving) continuous contracts. - -ADJUSTMENT (verified 2026-07 via test_adjustment.py): Norgate selects continuous -adjustment by SYMBOL SUFFIX, not the stock_price_adjustment_setting kwarg. The -BASE symbol '&ES' is UNADJUSTED (shows real calendar-spread gaps at each roll, -e.g. +146 pts at the 2026-06 Jun→Sep roll). '&ES_CCB' is BACK-ADJUSTED (gaps -stitched out). A close-based stop needs the gap-free series → we fetch _CCB. -""" -from __future__ import annotations # PEP 604 unions (dict | None) on Python 3.9 - -import datetime as dt -import re -from concurrent.futures import ThreadPoolExecutor, as_completed - -import numpy as np -import pandas as pd - -from .. import store -from ..registry import REGISTRY, all_symbols - -CCB_SUFFIX = "_CCB" # Norgate "Continuous Contract Back-adjusted" - -# Norgate database names (from norgatedata.databases()) that cotdata reads: the -# continuous series and the individual contracts used for volume reconstruction. -FINAL_DATABASES = ("Futures", "Continuous Futures") -# Default local-time cutoff after which Norgate's "Final" futures prices are in -# (≈ Continuous Futures Final at ~8:55pm ET per Norgate's update schedule). -DEFAULT_FINAL_CUTOFF = "20:55" -# Liquid continuous reference symbols for the data-driven finals gate: each trades every -# US futures session, so a newer settled bar on all of them means a session has landed. -_FINALS_REF_SYMBOLS = ("ES", "CL", "ZC") -# If roll-day overnight moves exceed this multiple of the normal-day median, the -# series looks UNADJUSTED (calendar-spread gaps not stitched). Self-calibrating -# per symbol, so it works across products with different spread magnitudes. -ROLL_GAP_RATIO_WARN = 1.5 - -_COLMAP = { - "Open": "Open", "High": "High", "Low": "Low", "Close": "Close", - "Volume": "Volume", "Open Interest": "Open Interest", - "Delivery Month": "Delivery Month", # kept → exact roll detection downstream -} - -MONTH_CODES = {'F': 1, 'G': 2, 'H': 3, 'J': 4, 'K': 5, 'M': 6, 'N': 7, 'Q': 8, 'U': 9, 'V': 10, 'X': 11, 'Z': 12} - -# Contract-spec fields fetched per symbol (everything in a metadata row except the -# Symbol/Norgate_Symbol identifiers). If Norgate returns nothing for ALL of these, -# the row is junk — skip it rather than persist an all-null spec row. -_SPEC_FIELDS = ("Name", "Exchange", "Group", "Contract Size", "Tick Size", - "Tick Value", "Point Value", "Currency", "Margin") - - -def _reconstruct_volume(internal_symbol: str, continuous_df: pd.DataFrame, adjustment: str, - full: bool = False) -> pd.DataFrame: - """Calculate FirstVolume, SecondVolume, and Volume_Reconstructed. - Uses incremental fetching based on the last successful Volume_Reconstructed date. - Returns continuous_df with the additive columns attached. - - full=True recomputes the ENTIRE history from scratch, ignoring the incremental - window. Needed when the reconstruction *logic* changes (not just new data): the - trailing-60-day window would otherwise leave old rows on the previous algorithm. - """ - import norgatedata - - # 1. Gap-aware incremental bounds - existing_df = store.read_prices(internal_symbol, adjustment) - last_date = pd.Timestamp("1970-01-01") - if not full and "Volume_Reconstructed" in existing_df.columns: - valid_dates = existing_df.dropna(subset=["Volume_Reconstructed"]).index - if len(valid_dates) > 0: - # Recompute trailing 60 days to catch late data & bridge partial failures - last_date = valid_dates.max() - pd.Timedelta(days=60) - - # Base Norgate symbol (e.g. "&ES" -> "ES") - base_sym = REGISTRY[internal_symbol].norgate.lstrip("&").split("_")[0] - - # 2. Find needed individual contracts - all_futures = norgatedata.database_symbols('Futures') - pattern = re.compile(rf"^{re.escape(base_sym)}-(\d{{4}})([FGHJKMNQUVXZ])$") - - needed_contracts = [] - for sym in all_futures: - m = pattern.match(sym) - if m: - year, month = int(m.group(1)), MONTH_CODES[m.group(2)] - expiry_date = pd.Timestamp(year=year, month=month, day=1) + pd.DateOffset(months=1) - if expiry_date >= last_date: - needed_contracts.append(sym) - - # 3. Fallback: if no individual contracts (e.g., crypto, ICE softs) - if not needed_contracts: - res = continuous_df.copy() - res["FirstVolume"] = np.nan - res["SecondVolume"] = np.nan - res["FirstContract"] = "" - res["SecondContract"] = "" - res["Volume_Reconstructed"] = res["Volume"] - res["Volume_Source"] = "raw" - return res - - # 4. Fetch OHLCV for needed contracts - frames = [] - with ThreadPoolExecutor(max_workers=10) as pool: - futs = { - pool.submit( - norgatedata.price_timeseries, - c, - padding_setting=norgatedata.PaddingType.NONE, - timeseriesformat="pandas-dataframe", - start_date=last_date.strftime("%Y-%m-%d") - ): c - for c in needed_contracts - } - for f in as_completed(futs): - c = futs[f] - try: - df_c = f.result() - if "Date" not in df_c.columns: - df_c = df_c.reset_index() - if not df_c.empty: - df_c["Symbol"] = c - frames.append(df_c[["Date", "Volume", "Symbol"]]) - except Exception as e: - print(f" ⚠️ Failed to fetch individual contract {c}: {e}") - - res = continuous_df.copy() - - if not frames: - res["FirstVolume"] = np.nan - res["SecondVolume"] = np.nan - res["FirstContract"] = "" - res["SecondContract"] = "" - res["Volume_Reconstructed"] = res["Volume"] - res["Volume_Source"] = "raw" - return res - - all_indiv = pd.concat(frames, ignore_index=True) - all_indiv['Date'] = pd.to_datetime(all_indiv['Date']).dt.tz_localize(None).dt.normalize() - - # 5. Contract identification: First / Second = the two HIGHEST-VOLUME contracts - # trading that day, NOT the two nearest by expiry. Products with serial months - # around a bi-monthly liquid cycle (e.g. GC, SI) carry almost no volume in the - # nearest serial month, so an expiry-order pick would sum near-empty contracts - # and badly understate true volume. Rank by descending volume; ties break by - # nearest expiry (columns are pre-sorted by expiry and the sort is stable); - # NaN (contract not trading that day) sorts last. - pivot = all_indiv.pivot(index="Date", columns="Symbol", values="Volume") - - def get_expiry(sym): - m = pattern.match(sym) - return pd.Timestamp(year=int(m.group(1)), month=MONTH_CODES[m.group(2)], day=1) - - sorted_cols = sorted(pivot.columns, key=get_expiry) - pivot = pivot[sorted_cols] - - vol_array = pivot.values - rank_key = np.where(np.isnan(vol_array), -np.inf, vol_array) - order = np.argsort(-rank_key, axis=1, kind='stable') - compressed = np.take_along_axis(vol_array, order, axis=1) - names_arr = np.array(pivot.columns) - - rec_df = pd.DataFrame(index=pivot.index) - - num_cols = compressed.shape[1] - if num_cols > 0: - rec_df['FirstVolume'] = compressed[:, 0] - rec_df['FirstContract'] = np.where(np.isnan(compressed[:, 0]), '', names_arr[order[:, 0]]) - else: - rec_df['FirstVolume'] = np.nan - rec_df['FirstContract'] = '' - - if num_cols > 1: - rec_df['SecondVolume'] = compressed[:, 1] - rec_df['SecondContract'] = np.where(np.isnan(compressed[:, 1]), '', names_arr[order[:, 1]]) - else: - rec_df['SecondVolume'] = np.nan - rec_df['SecondContract'] = '' - - rec_df['Volume_Reconstructed'] = rec_df['FirstVolume'].fillna(0) + rec_df['SecondVolume'].fillna(0) - rec_df.loc[rec_df['FirstVolume'].isna() & rec_df['SecondVolume'].isna(), 'Volume_Reconstructed'] = np.nan - rec_df['Volume_Source'] = "reconstructed" - - # 6. Merge the newly reconstructed subset into the existing history - for col in ["FirstVolume", "SecondVolume", "FirstContract", "SecondContract", "Volume_Reconstructed", "Volume_Source"]: - if col not in res.columns: - if col in existing_df.columns: - res[col] = existing_df[col] - else: - res[col] = "" if col in ("FirstContract", "SecondContract", "Volume_Source") else np.nan - - res.update(rec_df) - - common_idx = res.index.intersection(rec_df.index) - for col in ["FirstContract", "SecondContract"]: - res.loc[common_idx, col] = rec_df.loc[common_idx, col] - - mask = res['Volume_Reconstructed'].isna() - res.loc[mask, 'Volume_Reconstructed'] = res.loc[mask, 'Volume'] - res.loc[mask, 'Volume_Source'] = "raw" - - return res - - -def fetch(internal_symbol: str, adjustment: str = "backadj", start: str = "1970-01-01") -> pd.DataFrame: - """Fetch Norgate continuous bars: settlement close.""" - import norgatedata # imported lazily; only present on the Windows producer - ng_sym = REGISTRY[internal_symbol].norgate - if adjustment == "backadj": - ng_sym += CCB_SUFFIX - - df = norgatedata.price_timeseries( - ng_sym, - padding_setting=norgatedata.PaddingType.NONE, - timeseriesformat="pandas-dataframe", - start_date=start, - ) - df = df.rename(columns=_COLMAP) - keep = [c for c in _COLMAP.values() if c in df.columns] - out = df[keep].copy() - out.index = pd.to_datetime(out.index).tz_localize(None).normalize() - out.index.name = "Date" - return out.sort_index() - - -def _check_roll_gaps(internal_symbol: str, df: pd.DataFrame) -> bool: - """Warn if roll-day overnight moves ≫ normal-day moves — the signature of an - UNADJUSTED continuous. Returns True if it looks unadjusted. Self-calibrating: - compares each symbol's roll-day moves to its own non-roll baseline.""" - if "Delivery Month" not in df.columns or len(df) < 60: - return False - dm = df["Delivery Month"] - roll = dm.ne(dm.shift()) & dm.shift().notna() - if int(roll.sum()) < 8: - return False - overnight = (df["Close"] - df["Close"].shift(1)).abs() - roll_med = overnight[roll].median() - nonroll_med = overnight[~roll].median() - if nonroll_med and roll_med > ROLL_GAP_RATIO_WARN * nonroll_med: - print(f" ⚠️ {internal_symbol}: roll-day moves {roll_med:.1f} vs normal {nonroll_med:.1f} " - f"({roll_med / nonroll_med:.1f}x) — series looks UNADJUSTED. Expected the _CCB " - f"back-adjusted symbol; a close-based stop would false-trigger on roll gaps.") - return True - return False - - -def _to_naive_local(t): - """norgatedata returns tz-AWARE local datetimes (e.g. ...-04:00); normalize to - naive local so we can compare against a naive local cutoff. Naive inputs (older - norgatedata) are assumed already local and passed through.""" - if t is None: - return None - if t.tzinfo is not None: - t = t.astimezone().replace(tzinfo=None) # → local wall-clock, drop tzinfo - return t - - -def _finals_ready(db_times: dict, cutoff: str = DEFAULT_FINAL_CUTOFF, now=None): - """LEGACY wall-clock finals core, superseded by the data-driven gate - (:func:`_finals_ready_by_date` / :func:`finals_ready`). Retained for reference and - rollback; no CLI path uses it. It broke in production because a fixed clock cutoff - must sit below the earliest evening final yet above any daytime interim, and Norgate's - publish time drifts (see docs/design/finals_ready_data_driven.md). - - db_times maps database name → its last-update datetime (tz-aware or naive local, - or None). Ready when every database was refreshed at/after today's `cutoff` - (local HH:MM). Returns (ready: bool, detail: dict).""" - now = now or dt.datetime.now() - h, m = (int(x) for x in cutoff.split(":")) - cut = now.replace(hour=h, minute=m, second=0, microsecond=0) - detail, ready = {}, True - for db, t in db_times.items(): - detail[db] = t.isoformat() if t else None - tt = _to_naive_local(t) - if tt is None or tt < cut: - ready = False - detail["cutoff"] = cut.isoformat() - return ready, detail - - -def _finals_ready_by_date(norgate_last, store_last): - """Data-driven finals gate (pure core): ready when Norgate's latest continuous bar is - a NEWER completed session than the store already holds. - - Both args are dates (or datetimes, normalized to their date; or None). Norgate is - end-of-day and only publishes a session's bar once that session is complete, so a - bar date newer than the store's is a new *settled* session to capture. This needs no - trading calendar (weekends/holidays simply produce no new bar) and no wall-clock - cutoff (early publish → ready early; late publish → not there yet → a retry catches - it), which is what makes it immune to Norgate's publish-time drift. Returns - (ready: bool, detail: dict).""" - def _d(x): - if x is None: - return None - return x.date() if isinstance(x, dt.datetime) else x - - nl, sl = _d(norgate_last), _d(store_last) - detail = { - "norgate_last": nl.isoformat() if nl else None, - "store_last": sl.isoformat() if sl else None, - } - if nl is None: - return False, detail # Norgate has no bar to offer → defer - return (sl is None or nl > sl), detail - - -def _finals_ready_quorum(norgate_dates: dict, store_dates: dict): - """Combine per-reference results into the finals gate (pure/testable): ready only when - EVERY reference symbol has a newer settled bar in Norgate than the store already holds. - Requiring the whole quorum means a session is captured once and complete, and one - lagging reference cannot green-light a partial capture. Returns (ready, detail).""" - per, ready_all = {}, True - for sym in norgate_dates: - r, d = _finals_ready_by_date(norgate_dates.get(sym), store_dates.get(sym)) - per[sym] = {**d, "ready": r} - ready_all = ready_all and r - return ready_all, {"mode": "data", "per_symbol": per} - - -def _norgate_last_bar_date(sym: str): - """Latest continuous (back-adjusted) bar date Norgate holds for internal `sym`, or None. - Pulls a short trailing window (cheap) and takes the last index. Norgate is end-of-day - and only publishes a session's bar once settled, so this date advances exactly when a - new final session lands — the signal the gate keys on.""" - import norgatedata # Windows producer only - ng_sym = REGISTRY[sym].norgate + CCB_SUFFIX - start = (dt.date.today() - dt.timedelta(days=10)).isoformat() - df = norgatedata.price_timeseries( - ng_sym, - padding_setting=norgatedata.PaddingType.NONE, - timeseriesformat="pandas-dataframe", - start_date=start, - ) - if df is None or len(df) == 0: - return None - return pd.to_datetime(df.index[-1]).tz_localize(None).normalize().date() - - -def _store_last_bar_date(sym: str): - """Latest date already captured in the store for internal `sym` (back-adjusted), or - None if the store has never seen it. Read from the prices manifest — no price I/O.""" - prices = store.load_manifest().get("prices", {}) - ld = (prices.get(f"{sym}_backadj") or {}).get("last_date") - return pd.to_datetime(ld).date() if ld else None - - -def finals_ready(cutoff=None, now=None, ref_symbols=_FINALS_REF_SYMBOLS): - """Ready once Norgate has a NEWER settled continuous bar than the store, for a quorum of - liquid reference symbols (data-driven finals gate). Replaces the old wall-clock cutoff: - immune to Norgate's publish-time drift (early publish → ready early; late publish → not - there yet → a retry catches it) and needs no trading calendar (weekends and holidays - simply produce no new bar). See docs/design/finals_ready_data_driven.md. - - `cutoff` and `now` are accepted for backward compatibility and IGNORED — the clock gate - is deprecated. Returns (ready: bool, detail: dict).""" - _require_norgate_service() # NDU-down guard: norgatedata calls bare sys.exit otherwise - norgate_dates = {s: _norgate_last_bar_date(s) for s in ref_symbols} - store_dates = {s: _store_last_bar_date(s) for s in ref_symbols} - return _finals_ready_quorum(norgate_dates, store_dates) - - -def _norgate_covered(symbols): - """Resolve requested internal symbols to those Norgate actually carries. - - Yahoo-only markets (registry `norgate: null` — e.g. the MSCI MME/MFS indices - priced off ETF proxies) have no `&SYM_CCB` continuous series. Fetching them - errors on every field and, for metadata, silently writes null-filled rows, so - drop them here (with a note) rather than hitting Norgate for a symbol it can't - serve. The yfinance provider prices these instead.""" - requested = symbols or [s.internal for s in all_symbols()] - covered = [s for s in requested if REGISTRY[s].norgate] - skipped = [s for s in requested if not REGISTRY[s].norgate] - if skipped: - print(f" skipping {len(skipped)} symbol(s) with no Norgate coverage " - f"(priced elsewhere): {', '.join(skipped)}") - return covered - - -def _require_norgate_service() -> None: - """Fail fast, with a clear message and a normal exception, if the Norgate Data - Updater (NDU) service isn't reachable — BEFORE any fetch. - - Why this matters: norgatedata retries each data call 10x and then calls bare - ``sys.exit()``, which (a) exits with code 0, so a scheduled producer run looks - "successful" while writing nothing and never triggers the scheduler's retry, - and (b) raises SystemExit — not caught by the per-symbol ``except Exception`` — - so the whole run dies on the first symbol. ``norgatedata.status()`` is a safe - probe (haltonerror=False, maxretries=1 → returns False instead of exiting).""" - import norgatedata - try: - reachable = bool(norgatedata.status()) - except BaseException: # noqa: BLE001 — never let the probe itself take us down - reachable = False - if not reachable: - raise RuntimeError( - "Norgate Data service is not reachable — is the Norgate Data Updater " - "(NDU) running and authenticated? cotdata prices/metadata are produced " - "on Windows with NDU running. Aborting before fetch (non-zero exit so a " - "scheduler retries)." - ) - - -def update(symbols=None, full: bool = False) -> None: - """Fetch + write to the store for the given internal symbols (backadj and unadj). - - full=True forces a complete rebuild of the reconstructed-volume columns rather - than the trailing-60-day incremental update — use it after a reconstruction - logic change so old rows are recomputed under the new algorithm. - """ - import time - - from .. import status - - _require_norgate_service() # abort cleanly if NDU is down (see helper docstring) - syms = _norgate_covered(symbols) - prior = store.load_manifest().get("prices", {}) # to report per-symbol date deltas - t0 = time.time() - ok, failed, total_rows, newest = [], [], 0, None - for sym in syms: - try: - # 1. Back-Adjusted - out_backadj = fetch(sym, adjustment="backadj") - _check_roll_gaps(sym, out_backadj) # sanity: warn if backadj looks unadjusted - - # 2. Unadjusted (Raw calendar spreads) - out_unadj = fetch(sym, adjustment="unadj") - - # 3. Volume Reconstruction (Additive) - out_backadj = _reconstruct_volume(sym, out_backadj, "backadj", full=full) - out_unadj = _reconstruct_volume(sym, out_unadj, "unadj", full=full) - - store.write_prices(sym, "backadj", out_backadj, source="norgate") - store.write_prices(sym, "unadj", out_unadj, source="norgate") - - ok.append(sym) - total_rows += len(out_backadj) + len(out_unadj) - new = str(out_backadj.index.max().date()) if len(out_backadj) else "—" - newest = max(newest, new) if newest else new - was = (prior.get(f"{sym}_backadj") or {}).get("last_date") - delta = new if (was is None or was == new) else f"{was} -> {new}" - print(f"{sym:5s}: {len(out_backadj):6d} backadj, {len(out_unadj):6d} unadj [{delta}]") - except Exception as e: # noqa: BLE001 - failed.append((sym, e)) - print(f"{sym:5s}: FAILED — {e}") - - seconds = round(time.time() - t0, 1) - print(status.run_summary("prices update", ok, failed, total_rows, seconds, newest=newest)) - return { - "kind": "prices", "ok": ok, "failed": [(s, str(e)) for s, e in failed], - "symbols_failed": [s for s, _ in failed], "rows": total_rows, - "seconds": seconds, "newest": newest, - } - - -def get_symbol_metadata(internal_symbol: str) -> dict | None: - """Fetch contract specifications for a single continuous futures symbol.""" - import norgatedata # imported lazily - ng_sym = REGISTRY[internal_symbol].norgate + CCB_SUFFIX - - data = {'Symbol': internal_symbol, 'Norgate_Symbol': ng_sym} - try: - data['Name'] = norgatedata.security_name(ng_sym) - except Exception: - data['Name'] = None - try: - data['Exchange'] = norgatedata.exchange_name(ng_sym) - except Exception: - data['Exchange'] = None - try: - data['Group'] = norgatedata.classification_at_level( - ng_sym, - schemename='NorgateDataFuturesClassification', - classificationresulttype='Name', - level=1, - ) - except Exception: - data['Group'] = None - try: - data['Contract Size'] = norgatedata.point_value(ng_sym) - except Exception: - data['Contract Size'] = None - try: - data['Tick Size'] = norgatedata.tick_size(ng_sym) - except Exception: - data['Tick Size'] = None - - ts = data['Tick Size'] - cs = data['Contract Size'] - data['Tick Value'] = (ts * cs) if (ts is not None and cs is not None) else None - data['Point Value'] = cs - - try: - data['Currency'] = norgatedata.currency(ng_sym) - except Exception: - data['Currency'] = None - try: - data['Margin'] = norgatedata.margin(ng_sym) - except Exception: - data['Margin'] = None - - return data - - -def update_metadata(symbols=None) -> None: - """Fetch and write contract specifications (metadata) to the store. - - A scoped run (`symbols` given) UPSERTS by Symbol into the existing - contract_specs table — rows for markets NOT in the request are preserved - (contract specs share one table, so a plain write would drop them). With no - `symbols`, regenerate the full registry and replace the table. - """ - import concurrent.futures - scoped = symbols is not None - _require_norgate_service() # abort cleanly if NDU is down (see helper docstring) - syms = _norgate_covered(symbols) - - print(f"Fetching metadata for {len(syms)} symbols...") - metadata_rows = [] - - with concurrent.futures.ThreadPoolExecutor(max_workers=10) as pool: - futs = {pool.submit(get_symbol_metadata, s): s for s in syms} - for f in concurrent.futures.as_completed(futs): - result = f.result() - if not result: - continue - # A covered symbol whose specs all came back None is a transient Norgate - # failure, not real data — skip rather than persist a null row (and, on a - # scoped upsert, rather than overwrite good existing specs with nulls). - if all(result.get(k) is None for k in _SPEC_FIELDS): - print(f" ⚠️ {result.get('Symbol')}: all specs empty (Norgate " - f"returned nothing) — skipping to avoid a null row") - continue - metadata_rows.append(result) - - if metadata_rows: - df = pd.DataFrame(metadata_rows) - # Ensure consistent column ordering and sorting - df = df.sort_values("Symbol").reset_index(drop=True) - if scoped: - store.upsert_metadata(df, source="norgate") - print(f"Upserted metadata for {len(df)} symbols -> store (unlisted markets preserved)") - else: - store.write_metadata(df, source="norgate") - print(f"Successfully wrote metadata for {len(df)} symbols -> store") - else: - print("No metadata fetched.") diff --git a/src/cotdata/providers/yfinance.py b/src/cotdata/providers/yfinance.py deleted file mode 100644 index e31180a..0000000 --- a/src/cotdata/providers/yfinance.py +++ /dev/null @@ -1,70 +0,0 @@ -"""yfinance price provider — free Yahoo Finance OHLCV for registry symbols that carry -a ``yahoo`` ticker. - -For markets Norgate/databento don't cover (e.g. MSCI EM / EAFE, priced via the EEM / -EFA ETF proxies). Research-grade: Yahoo is a free, unofficial feed — expect occasional -gaps, silent revisions, and API breakage; not a production replacement for Norgate. - -Writes the same Open/High/Low/Close/Volume frame with a tz-naive DatetimeIndex named -``Date`` that the store's other price providers use, so ``cotdata.get_prices`` stays -source-agnostic. ETF/spot proxies have no futures roll, so backadj == unadj (both are -written, since consumers ask for ``backadj``). -""" -from __future__ import annotations - -import pandas as pd - -from .. import store -from ..registry import all_symbols, default_price_source, resolve_source - - -def _fetch(ticker: str) -> pd.DataFrame: - import yfinance as yf - raw = yf.download(ticker, period="max", auto_adjust=True, - progress=False, threads=False) - if raw is None or raw.empty: - return pd.DataFrame() - # yfinance returns a (field, ticker) column MultiIndex even for a single symbol. - if isinstance(raw.columns, pd.MultiIndex): - raw.columns = raw.columns.get_level_values(0) - keep = [c for c in ("Open", "High", "Low", "Close", "Volume") if c in raw.columns] - df = raw[keep].copy() - df.index = pd.to_datetime(df.index).tz_localize(None) - df.index.name = "Date" - return df.dropna(subset=["Close"]).sort_index() - - -def update(symbols=None) -> dict: - """Fetch Yahoo OHLCV for registry symbols that RESOLVE to yfinance on this - deployment (see registry.resolve_source): markets the default vendor can't serve - (e.g. MSCI ETF proxies always; ICE softs when the default is databento) plus any - explicit ``price_source: yfinance`` override. Keyed on $COTDATA_PRICE_SOURCE, so - the same softs stay on Norgate locally and fall to Yahoo on a databento server. - Pass ``symbols`` to scope. Returns {kind, ok, wrote}.""" - default = default_price_source() - targets = [s for s in all_symbols() - if s.yahoo and resolve_source(s, default) == "yfinance" - and (symbols is None or s.internal in symbols)] - if not targets: - print("yfinance: no registry symbols with a 'yahoo' ticker" - + (f" among {symbols}" if symbols else "")) - return {"kind": "prices_yahoo", "ok": True, "wrote": 0} - - wrote, failed = 0, 0 - for s in targets: - try: - df = _fetch(s.yahoo) - except Exception as e: # noqa: BLE001 — yfinance/network is flaky by nature - print(f"{s.internal}: yfinance fetch failed ({s.yahoo}) — {e}") - failed += 1 - continue - if df.empty: - print(f"{s.internal}: yfinance returned no data ({s.yahoo})") - failed += 1 - continue - for adj in ("backadj", "unadj"): - store.write_prices(s.internal, adj, df, source="yahoo") - wrote += 1 - print(f"{s.internal}: {len(df):5d} bars ({s.yahoo}) " - f"{df.index.min().date()}..{df.index.max().date()} -> store") - return {"kind": "prices_yahoo", "ok": failed == 0, "wrote": wrote} diff --git a/src/cotdata/registry.py b/src/cotdata/registry.py index e295149..4769afd 100644 --- a/src/cotdata/registry.py +++ b/src/cotdata/registry.py @@ -173,6 +173,14 @@ def by_asset_class(asset_class: str) -> List[Symbol]: # registry: (1) a deployment default (COTDATA_PRICE_SOURCE), (2) per-symbol capability # (which vendors carry a series — the norgate/databento/yahoo mappings), and (3) an # optional per-symbol override. See ADR-0006. +# +# ADR-0007 NOTE. Only the `databento` mapping has a producer left in this package — +# `norgate` and `yahoo` moved to `marketdata`, which resolves them against its OWN +# registry. They stay here because this file is a CAPABILITY map that a deployment may +# share between the two packages via COTDATA_REGISTRY, and because deleting a column +# from a shared registry breaks the other reader. So `resolve_source` can still name a +# vendor this package cannot produce: that is a statement about the market, not a +# promise that `cotdata-update` will fetch it. def _can_serve(sym: Symbol, source: str) -> bool: """Whether `source` has a price series for `sym` (its vendor mapping is present).""" diff --git a/src/cotdata/store.py b/src/cotdata/store.py index 35b034d..da2d555 100644 --- a/src/cotdata/store.py +++ b/src/cotdata/store.py @@ -28,37 +28,15 @@ def _atomic_write_parquet(df: pd.DataFrame, path: Path) -> None: os.remove(tmp) -# ── Metadata ────────────────────────────────────────────────────────────── -# Unlike prices/COT (one parquet per symbol), contract specs live in ONE table -# keyed by Symbol. So a *scoped* refresh must upsert (see upsert_metadata) — a -# plain write_metadata would replace the whole table and drop unlisted markets. -def write_metadata(df: pd.DataFrame, source: str = "norgate") -> None: - _atomic_write_parquet(df, config.metadata_dir() / "contract_specs.parquet") - _touch_manifest("metadata", "contract_specs", df, source) - - -def upsert_metadata(df: pd.DataFrame, source: str = "norgate") -> None: - """Merge `df` into the existing contract_specs table by ``Symbol``: rows for - symbols present in `df` are replaced/added; rows for symbols NOT in `df` are - kept. Use for a scoped (subset-of-symbols) refresh so it never drops the - contract specs of markets that weren't in the request. Use write_metadata to - replace the whole table (a full registry regeneration).""" - existing = read_metadata() - if not existing.empty and "Symbol" in existing.columns: - keep = existing[~existing["Symbol"].isin(df["Symbol"])] - merged = pd.concat([keep, df], ignore_index=True) - else: - merged = df - merged = merged.sort_values("Symbol").reset_index(drop=True) - write_metadata(merged, source=source) - - -def read_metadata() -> pd.DataFrame: - p = config.metadata_dir() / "contract_specs.parquet" - return pd.read_parquet(p) if p.exists() else pd.DataFrame() - - # ── Prices ──────────────────────────────────────────────────────────────── +# ADR-0007 makes this package COT-only, and the CONSUMER bar API (get_prices, +# roll_dates, the derived propadj tier) is gone — read bars from `marketdata`. +# +# These two survive because the databento provider does not have a marketdata +# equivalent yet and still writes here (ADR-0006: an independently-built +# alternative to Norgate, and the source for the intraday news-failure work). +# Keeping the write path without the read path would make its output unreadable +# through the library, so the store-level pair stays; only the consumer API left. def write_prices(symbol: str, adjustment: str, df: pd.DataFrame, source: str) -> None: _atomic_write_parquet(df, config.prices_dir() / f"{symbol}_{adjustment}.parquet") _touch_manifest("prices", f"{symbol}_{adjustment}", df, source) @@ -123,6 +101,12 @@ def read_cot_supplemental(name: str) -> pd.DataFrame: # the CFTC producer writes the `cot` half, the price producers write the `prices` # half, and they never touch the same manifest file. Listing every domain here (and # refusing unknown ones) is what stops a new domain quietly joining the wrong side. +# +# `metadata` (futures contract specs) is still DECLARED but nothing writes it any +# more: ADR-0007 moved contract specs to marketdata along with the bars. It stays +# on the map because stores written before that move carry `metadata` entries, and +# an undeclared domain is skipped by migrate_manifests() — which would strand those +# entries in the legacy aggregate forever. Read-only legacy; do not add writers. HALVES = ("cot", "prices") _DOMAIN_HALF = { "prices": "prices", @@ -148,7 +132,10 @@ def half_for(kind: str) -> str: def _empty_manifest() -> dict: - return {"schema_version": config.SCHEMA_VERSION, "metadata": {}, "prices": {}, + # No `metadata` key: a store created from here on never gets contract specs + # (they live in marketdata now). Legacy stores that have them still read fine — + # load_manifest() overlays whatever domains it finds on disk. + return {"schema_version": config.SCHEMA_VERSION, "prices": {}, "cot_legacy": {}, "cot_disagg": {}, "cot_tff": {}, "cot_supplemental": {}} diff --git a/src/cotdata/update.py b/src/cotdata/update.py index 62f507b..2c682e5 100644 --- a/src/cotdata/update.py +++ b/src/cotdata/update.py @@ -1,9 +1,14 @@ -"""Producer CLI: cotdata-update --prices --symbols ES NQ - cotdata-update --cot-all +"""Producer CLI: cotdata-update --cot-all + cotdata-update --build-databento --symbols ES NQ cotdata-update --check # read-only store status cotdata-update --reconcile # prune stale manifest ghosts -Writes to $COTDATA_STORE. Schedule prices nightly (after the Norgate Data -Updater) and COT weekly (Friday, after the CFTC release).""" +Writes to $COTDATA_STORE. Schedule COT weekly (Friday, after the CFTC release). + +ADR-0007 moved bar production out of this package: the Norgate and Yahoo price +producers, and the consumer bar API, now live in `marketdata` — run +`marketdata-update --bars` on the nightly job. What remains on the price side is +databento, which has no marketdata equivalent yet (ADR-0006: an independently +built alternative to Norgate, and the source for the intraday work).""" import argparse import datetime as _dt @@ -16,8 +21,7 @@ # exist to contain. Read-only actions (--check, --reconcile) belong to both. _HALF_ACTIONS = { "cot": ("cot_legacy", "cot_disagg", "cot_tff", "cot_supplemental", "cot_all"), - "prices": ("prices", "metadata", "prices_yahoo", "ingest_databento", - "build_databento"), + "prices": ("ingest_databento", "build_databento"), } @@ -32,13 +36,18 @@ def _reject_other_half(parser, args, half: str) -> None: f"cotdata-update to run both from one machine.") -def main(argv=None, half=None) -> None: +# Parser dests that are NOT producer actions: read-only or one-shot maintenance, and +# modifiers on another flag. Everything else must be classified into a half above — +# test_every_action_flag_is_assigned_to_a_half reads both lists to enforce it, so a new +# action flag fails the suite until someone decides which side of the seam it is on. +_NON_ACTIONS = frozenset({ + "help", "symbols", "check", "migrate_manifests", "reconcile", + "reconcile_databento", "windowed_n1_stats", "batch", +}) + + +def _parser() -> argparse.ArgumentParser: p = argparse.ArgumentParser(description="cotdata producer — fetch sources into the store.") - p.add_argument("--prices", action="store_true", help="Update Norgate price bars (Windows).") - p.add_argument("--metadata", action="store_true", help="Update Norgate contract metadata (Windows).") - p.add_argument("--prices-yahoo", action="store_true", - help="Update prices from Yahoo Finance for registry symbols that resolve to " - "yfinance on this deployment (cross-platform; research-grade).") p.add_argument("--ingest-databento", action="store_true", help="Databento Stage 1 (paid API, cross-platform): fetch raw .n.0/.n.1 " "ohlcv-1d + statistics into the append-only raw store ($COTDATA_DATABENTO_RAW). " @@ -70,18 +79,6 @@ def main(argv=None, half=None) -> None: p.add_argument("--cot-all", action="store_true", help="Update all COT pipelines (Legacy, Disagg, TFF, Supplemental).") p.add_argument("--symbols", nargs="+", default=None, help="Internal symbols; default = all in registry.") - p.add_argument("--full", action="store_true", - help="Full rebuild of reconstructed volume (ignore the incremental " - "60-day window). Use after a reconstruction-logic change.") - p.add_argument("--require-final", action="store_true", - help="For --prices: only fetch once Norgate has a NEWER settled continuous " - "bar than the store already holds (a new session has landed). " - "Otherwise defer with a non-zero exit so a scheduler retries. Immune " - "to Norgate's publish-time drift; needs no cutoff or calendar.") - p.add_argument("--final-cutoff", default="20:55", metavar="HH:MM", - help="DEPRECATED and ignored: the finals gate is now data-driven " - "(newer-bar-than-store), not a wall-clock cutoff. Accepted so " - "existing schedulers do not break.") p.add_argument("--check", action="store_true", help="Print store status (row counts, newest data, staleness) from " "the manifest and exit. Read-only, cross-platform, no network.") @@ -99,6 +96,11 @@ def main(argv=None, half=None) -> None: "entries whose file is missing (so a restart does not skip them as " "'already current' and leave a silent hole). Local files only, no " "API. Exit.") + return p + + +def main(argv=None, half=None) -> None: + p = _parser() args = p.parse_args(argv) if half: _reject_other_half(p, args, half) @@ -161,45 +163,15 @@ def main(argv=None, half=None) -> None: print(" symbols: " + ", ".join(syms)) return - if not (args.prices or args.metadata or args.prices_yahoo or args.ingest_databento - or args.build_databento or args.cot_legacy or args.cot_disagg or args.cot_tff - or args.cot_supplemental or args.cot_all): - p.error("nothing to do — pass --check, --prices, --prices-yahoo, --ingest-databento, " - "--build-databento, --metadata, --cot-legacy, --cot-disagg, --cot-tff, " - "--cot-supplemental, or --cot-all") - - if args.prices or args.metadata: - from .providers import norgate + if not (args.ingest_databento or args.build_databento or args.cot_legacy + or args.cot_disagg or args.cot_tff or args.cot_supplemental or args.cot_all): + p.error("nothing to do — pass --check, --ingest-databento, --build-databento, " + "--cot-legacy, --cot-disagg, --cot-tff, --cot-supplemental, or " + "--cot-all. Norgate/Yahoo bars moved to marketdata (ADR-0007): use " + "'marketdata-update --bars'.") kinds = [] - last_run = None failed_kinds = [] # domains that hard-failed → non-zero exit so a scheduler retries - deferred = [] # work skipped because inputs aren't ready yet (also non-zero exit) - if args.prices: - ready = True - if args.require_final: - ready, detail = norgate.finals_ready() - if not ready: - print(f"prices: no new settled Norgate bar yet (--require-final) " - f"— deferring. {detail}") - deferred.append("prices") - if ready: - last_run = norgate.update(symbols=args.symbols, full=args.full) - kinds.append("prices") - if last_run and last_run.get("symbols_failed"): - failed_kinds.append("prices") - - if args.metadata: - norgate.update_metadata(symbols=args.symbols) - kinds.append("metadata") - - if args.prices_yahoo: - from .providers import yfinance as yprov - r = yprov.update(symbols=args.symbols) - kinds.append("prices_yahoo") - if not (r or {}).get("ok", True): - failed_kinds.append("prices_yahoo") - if args.ingest_databento: from .providers import databento if args.batch: @@ -256,21 +228,19 @@ def main(argv=None, half=None) -> None: # Structured heartbeat for downstream tools: rebuild status.json from the now- # updated manifest. Pollers detect new data via newest_data[]. from . import status - run = dict(last_run or {}) - run["kinds"] = kinds - run["failed"] = failed_kinds - run["deferred"] = deferred - run["at"] = _dt.datetime.utcnow().isoformat(timespec="seconds") + "Z" + # `deferred` is gone with the Norgate finals gate: no source left here has a + # "not published yet" state — the CFTC zips and the databento raw store are + # either reachable or they are not. The gate itself moved to + # `marketdata-update --bars --require-final`. + run = {"kinds": kinds, "failed": failed_kinds, + "at": _dt.datetime.utcnow().isoformat(timespec="seconds") + "Z"} path = status.write_status_file(last_run=run) print(f"status written -> {path}") - # Exit non-zero so Task Scheduler / cron retries, either on a hard failure - # (source unreachable) or when --require-final deferred because the Finals - # aren't in yet. Ordinary "no new data yet" is NOT a failure. - if failed_kinds or deferred: - parts = ([f"failed: {', '.join(failed_kinds)}"] if failed_kinds else []) + \ - ([f"deferred: {', '.join(deferred)}"] if deferred else []) - raise SystemExit("cotdata-update: " + " | ".join(parts)) + # Exit non-zero on a hard failure (source unreachable) so Task Scheduler / cron + # retries. Ordinary "no new data yet" is NOT a failure. + if failed_kinds: + raise SystemExit(f"cotdata-update: failed: {', '.join(failed_kinds)}") if __name__ == "__main__": main() @@ -282,5 +252,6 @@ def main_cot(argv=None) -> None: def main_prices(argv=None) -> None: - """`cotdata-prices`: the price half. Norgate needs Windows with NDU running.""" + """`cotdata-prices`: the price half, now databento only (ADR-0007 moved Norgate + and Yahoo to `marketdata`). Cross-platform; needs DATABENTO_API_KEY to ingest.""" main(argv, half="prices") diff --git a/tests/test_adjustment.py b/tests/test_adjustment.py deleted file mode 100644 index bb7ab9b..0000000 --- a/tests/test_adjustment.py +++ /dev/null @@ -1,111 +0,0 @@ -"""Diagnose Norgate price adjustment: is plain '&ES' back-adjusted or unadjusted? - -Back-adjusted continuous = gap-free at rolls (contracts stitched with an offset). -Unadjusted continuous = shows the real calendar-spread GAP at every roll. - -A close-based CMR stop on UNADJUSTED data would false-trigger on roll gaps, so we -need the back-adjusted series. This checks '&ES' vs '&ES_CCB' and measures the -overnight move at roll dates (where Delivery Month changes). - -Run on Windows: python test_adjustment.py -""" -import pandas as pd - -try: - import norgatedata -except ImportError: - # Fail cleanly under pytest collection on Mac/Linux instead of crashing. - norgatedata = None - -pd.set_option("display.width", 140) -pd.set_option("display.max_columns", 20) - - -def fetch(sym): - df = norgatedata.price_timeseries( - sym, - padding_setting=norgatedata.PaddingType.NONE, - timeseriesformat="pandas-dataframe", - start_date="2015-01-01", - ) - df.index = pd.to_datetime(df.index).tz_localize(None).normalize() - return df - - -def roll_gap_report(label, df): - """At each Delivery Month change, report the overnight Close move vs typical range.""" - if "Delivery Month" not in df.columns: - print(f" {label}: no Delivery Month column — can't locate rolls") - return - dm = df["Delivery Month"] - roll_mask = dm.ne(dm.shift()) & dm.shift().notna() - rolls = df.index[roll_mask] - prev_close = df["Close"].shift(1) - overnight = (df["Close"] - prev_close).abs() - daily_range = (df["High"] - df["Low"]) - typ_range = daily_range.median() - roll_gaps = overnight[roll_mask] - print(f" {label}: {len(rolls)} rolls | median roll overnight move = {roll_gaps.median():.2f} pts " - f"({roll_gaps.median() / typ_range:.2f}x typical daily range {typ_range:.2f})") - print(f" max roll move = {roll_gaps.max():.2f} pts on {roll_gaps.idxmax().date() if len(roll_gaps) else 'n/a'}") - # show a few biggest roll gaps - biggest = roll_gaps.sort_values(ascending=False).head(5) - for dt, g in biggest.items(): - print(f" {dt.date()}: overnight {g:.2f} pts (Close {prev_close[dt]:.2f} -> {df['Close'][dt]:.2f}, DM {int(dm.shift()[dt])}->{int(dm[dt])})") - - -def main(): - if norgatedata is None: - print("Error: norgatedata package is not installed. This script must be run on the Windows producer machine.") - return - - print("=" * 70) - print("1. Which ES continuous symbols exist?") - print("=" * 70) - try: - cont = norgatedata.database_symbols("Continuous Futures") - es_syms = [s for s in cont if s.lstrip("&").upper().startswith("ES")] - print(" ES-related continuous symbols:", es_syms) - except Exception as e: - print(" database_symbols failed:", e) - - print() - print("=" * 70) - print("2. Fetch '&ES' and '&ES_CCB', compare") - print("=" * 70) - a = fetch("&ES") - print(f" &ES : {len(a)} bars {a.index.min().date()}..{a.index.max().date()}") - try: - b = fetch("&ES_CCB") - print(f" &ES_CCB : {len(b)} bars {b.index.min().date()}..{b.index.max().date()}") - join = a[["Close"]].join(b[["Close"]], rsuffix="_CCB", how="inner") - diff = (join["Close"] - join["Close_CCB"]).abs() - print(f" Close differs on {int((diff > 0.01).sum())}/{len(join)} shared days " - f"(max diff {diff.max():.2f} pts)") - print(" → if they differ, they are DIFFERENT series (one adj, one unadj)") - have_ccb = True - except Exception as e: - print(f" &ES_CCB : NOT AVAILABLE — {e}") - have_ccb = False - - print() - print("=" * 70) - print("3. Roll-gap test (the decisive check)") - print("=" * 70) - print(" Back-adjusted → roll overnight move ≈ typical daily range (stitched).") - print(" Unadjusted → roll overnight move ≫ typical (real calendar-spread gap).") - print() - roll_gap_report("&ES ", a) - if have_ccb: - roll_gap_report("&ES_CCB", b) - - print() - print("=" * 70) - print("VERDICT") - print("=" * 70) - print(" If '&ES' roll moves are ≫ typical range → '&ES' is UNADJUSTED; switch the") - print(" producer's backadj symbol to '&ES_CCB'. If '&ES' roll moves ≈ typical →") - print(" '&ES' is already back-adjusted and the current producer is fine.") - -if __name__ == "__main__": - main() diff --git a/tests/test_cli_exit.py b/tests/test_cli_exit.py index 2cd3583..dc52dbc 100644 --- a/tests/test_cli_exit.py +++ b/tests/test_cli_exit.py @@ -29,32 +29,38 @@ def test_exits_zero_on_cot_success(tmp_path, monkeypatch): update.main() # must not raise SystemExit -def test_exits_nonzero_when_prices_have_failures(tmp_path, monkeypatch): - _argv(monkeypatch, tmp_path, "--prices") +def test_exits_nonzero_when_the_databento_build_fails(tmp_path, monkeypatch): + """The price half is databento only now: ADR-0007 moved the Norgate and Yahoo + producers (and the --require-final finals gate that covered them) to marketdata. + What this file protects is unchanged — a scheduler must be able to tell a failed + run from a quiet one.""" + _argv(monkeypatch, tmp_path, "--build-databento") from cotdata import update - with mock.patch("cotdata.providers.norgate.update", - return_value={"kind": "prices", "symbols_failed": ["GC"], "ok": []}): + with mock.patch("cotdata.providers.databento.build", + return_value={"kind": "build_databento", "ok": False, "wrote": 0}): with pytest.raises(SystemExit) as ei: update.main() assert ei.value.code not in (0, None) -def test_require_final_defers_when_not_ready(tmp_path, monkeypatch): - _argv(monkeypatch, tmp_path, "--prices", "--require-final") +def test_exits_zero_on_databento_build_success(tmp_path, monkeypatch): + _argv(monkeypatch, tmp_path, "--build-databento") from cotdata import update - with mock.patch("cotdata.providers.norgate.finals_ready", return_value=(False, {"Futures": None})), \ - mock.patch("cotdata.providers.norgate.update") as m_update: - with pytest.raises(SystemExit) as ei: - update.main() - assert ei.value.code not in (0, None) # non-zero -> scheduler retries - m_update.assert_not_called() # did NOT capture interim prices + with mock.patch("cotdata.providers.databento.build", + return_value={"kind": "build_databento", "ok": True, "wrote": 3}): + update.main() # must not raise SystemExit -def test_require_final_runs_when_ready(tmp_path, monkeypatch): - _argv(monkeypatch, tmp_path, "--prices", "--require-final") +def test_retired_price_flags_are_refused_not_ignored(tmp_path, monkeypatch): + """A scheduler line still carrying --prices must fail loudly. + + argparse rejects an unknown flag, so this is really a guard against quietly + re-adding one as a no-op alias: a nightly job that keeps exiting 0 while + fetching nothing is a store that silently stops being updated. + """ from cotdata import update - with mock.patch("cotdata.providers.norgate.finals_ready", return_value=(True, {})), \ - mock.patch("cotdata.providers.norgate.update", - return_value={"kind": "prices", "symbols_failed": [], "ok": ["ES"]}) as m_update: - update.main() # must not raise - m_update.assert_called_once() + for flag in ("--prices", "--prices-yahoo", "--metadata", "--require-final"): + _argv(monkeypatch, tmp_path, flag) + with pytest.raises(SystemExit) as ei: + update.main() + assert ei.value.code not in (0, None) diff --git a/tests/test_databento_build.py b/tests/test_databento_build.py index 555d935..757a675 100644 --- a/tests/test_databento_build.py +++ b/tests/test_databento_build.py @@ -1,15 +1,20 @@ """Stage-2 databento build (ADR-0006): additive back-adjustment from the raw store. Populates a raw store directly (as ingest would leave it), runs build(), and reads -the result back through the real consumer API (cotdata.get_prices) to verify unadj, -settlement override, Open Interest, and the Norgate-style additive back-adjustment. +the result back out of the store to verify unadj, settlement override, Open Interest, +and the Norgate-style additive back-adjustment. + +Reads via `store.read_prices` rather than a consumer bar API: ADR-0007 moved that API +to `marketdata` and left only the store-level pair databento writes through. The two +differ only in normalisation (`get_prices` sorted the index and named it 'Date'), and +build() writes a sorted DatetimeIndex, so what is asserted here is unchanged. """ from pathlib import Path import pandas as pd import pytest -import cotdata +from cotdata import store from cotdata.providers.databento import build @@ -63,8 +68,8 @@ def test_build_back_adjusts_the_roll_gap(stores): res = build(["ES"]) assert res["ok"] and res["wrote"] == 1 - unadj = cotdata.get_prices("ES", adjustment="unadj") - backadj = cotdata.get_prices("ES", adjustment="backadj") + unadj = store.read_prices("ES", "unadj") + backadj = store.read_prices("ES", "backadj") # unadj keeps the raw front prices (roll gap intact: 102 -> 110). assert list(unadj["Close"]) == [100, 101, 102, 110, 111, 112] @@ -88,8 +93,8 @@ def test_build_detects_rolls_from_instrument_id_not_symbol(stores): sym="ES.n.1", instrument_id=[20, 20, 20, 30, 30, 30]) build(["ES"]) - unadj = cotdata.get_prices("ES", adjustment="unadj") - backadj = cotdata.get_prices("ES", adjustment="backadj") + unadj = store.read_prices("ES", "unadj") + backadj = store.read_prices("ES", "backadj") # The constant `symbol` alias would find no rolls; instrument_id finds the d3 roll, # gap = 105 - 102 = +3, applied to the pre-roll segment. assert list(unadj["Close"]) == [100, 101, 102, 110, 111, 112] @@ -106,8 +111,8 @@ def test_build_uses_settlement_and_open_interest(stores): _write_stats(raw, "ES", ".n.1", dates, settle=[103.5, 104.5, 105.5, 113.5, 114.5, 115.5]) build(["ES"]) - unadj = cotdata.get_prices("ES", adjustment="unadj") - backadj = cotdata.get_prices("ES", adjustment="backadj") + unadj = store.read_prices("ES", "unadj") + backadj = store.read_prices("ES", "backadj") # Close is the settlement, not the ohlcv last trade; OI comes from stat_type 9. assert list(unadj["Close"]) == [100.5, 101.5, 102.5, 110.5, 111.5, 112.5] @@ -123,8 +128,8 @@ def test_build_no_rolls_leaves_series_unadjusted(stores, capsys): _write_ohlcv(raw, "ES", ".n.1", dates, [110, 111, 112, 113, 114], ["B"] * 5) build(["ES"]) - unadj = cotdata.get_prices("ES", adjustment="unadj") - backadj = cotdata.get_prices("ES", adjustment="backadj") + unadj = store.read_prices("ES", "unadj") + backadj = store.read_prices("ES", "backadj") assert list(backadj["Close"]) == list(unadj["Close"]) # no roll → no adjustment assert "no rolls detected" in capsys.readouterr().out @@ -140,7 +145,7 @@ def test_build_applies_norgate_unit_scale(stores): _write_stats(raw, "SI", ".n.0", dates, oi=[7000] * 4) build(["SI"]) - unadj = cotdata.get_prices("SI", adjustment="unadj") + unadj = store.read_prices("SI", "unadj") assert list(unadj["Close"]) == [2500.0, 2510.0, 2520.0, 2530.0] # x100 -> cents assert unadj["High"].iloc[0] == (25.0 + 0.5) * 100 assert list(unadj["Open Interest"]) == [7000] * 4 # counts unscaled @@ -154,7 +159,7 @@ def test_build_leaves_unscaled_symbol_in_native_units(stores): dates = pd.date_range("2020-01-01", periods=3, freq="D") _write_ohlcv(raw, "ES", ".n.0", dates, [4000.0, 4001.0, 4002.0], "A") build(["ES"]) - assert list(cotdata.get_prices("ES", adjustment="unadj")["Close"]) == [4000.0, 4001.0, 4002.0] + assert list(store.read_prices("ES", "unadj")["Close"]) == [4000.0, 4001.0, 4002.0] def test_build_skips_symbol_missing_from_raw_store(stores): @@ -213,7 +218,7 @@ def stats(settle): def test_windowed_n1_stats_matches_full_backadj(tmp_path, monkeypatch): - import cotdata + from cotdata import store from cotdata.providers.databento import ingest dates = pd.date_range("2020-01-01", periods=6) # roll at 2020-01-03 (id 10→20) @@ -225,7 +230,7 @@ def run(tag, window): ingest(symbols=["ES"], client=client, end="2020-01-07", cold_start="2020-01-01", n1_stats_window=window) build(["ES"]) - return cotdata.get_prices("ES", adjustment="backadj"), client + return store.read_prices("ES", "backadj"), client full_bad, full_client = run("full", None) win_bad, win_client = run("win", 1) diff --git a/tests/test_manifest_seam.py b/tests/test_manifest_seam.py index 66a440d..5293921 100644 --- a/tests/test_manifest_seam.py +++ b/tests/test_manifest_seam.py @@ -144,7 +144,7 @@ def test_cot_entry_point_refuses_price_actions(store_env, capsys): racing the first, which is exactly what the split manifests exist to contain.""" from cotdata import update with pytest.raises(SystemExit): - update.main_cot(["--prices"]) + update.main_cot(["--build-databento"]) assert "belong(s) to the prices half" in capsys.readouterr().err @@ -169,9 +169,8 @@ def test_combined_entry_point_applies_no_half_restriction(): from cotdata import update parser = argparse.ArgumentParser() - both = argparse.Namespace(prices=True, cot_all=True, metadata=False, - prices_yahoo=False, ingest_databento=False, - build_databento=False, cot_legacy=False, + both = argparse.Namespace(build_databento=True, cot_all=True, + ingest_databento=False, cot_legacy=False, cot_disagg=False, cot_tff=False) # Each scoped half rejects the other's action ... for half in ("cot", "prices"): @@ -183,13 +182,17 @@ def test_combined_entry_point_applies_no_half_restriction(): def test_every_action_flag_is_assigned_to_a_half(): - """A new action must be classified, or the entry points silently allow it.""" + """A new action must be classified, or the entry points silently allow it. + + Read off the PARSER, not a list copied beside it. The copied list was the bug: + it stayed green through this change while naming three flags that no longer + exist, so it could not have caught a fourth being added either. Now a new flag + fails here until it is put on one side of the seam or declared a non-action. + """ from cotdata import update + flags = {a.dest for a in update._parser()._actions} - update._NON_ACTIONS assigned = set(update._HALF_ACTIONS["cot"]) | set(update._HALF_ACTIONS["prices"]) - actions = {"prices", "metadata", "prices_yahoo", "ingest_databento", - "build_databento", "cot_legacy", "cot_disagg", "cot_tff", - "cot_supplemental", "cot_all"} - assert actions == assigned + assert flags == assigned # ── dropping the legacy aggregate ───────────────────────────────────────── diff --git a/tests/test_norgate_provider.py b/tests/test_norgate_provider.py deleted file mode 100644 index 4d469fb..0000000 --- a/tests/test_norgate_provider.py +++ /dev/null @@ -1,520 +0,0 @@ -# Mock norgatedata so we can import and test norgate on any OS -import datetime as _dt -import sys -import types -from unittest import mock - -import numpy as np -import pandas as pd -import pytest - -mock_norgatedata = types.ModuleType("norgatedata") -mock_norgatedata.PaddingType = mock.Mock() -mock_norgatedata.PaddingType.NONE = "NONE" -mock_norgatedata.status = mock.Mock(return_value=True) # NDU reachable (preflight) -sys.modules["norgatedata"] = mock_norgatedata - -from cotdata.providers import norgate # noqa: E402 (import after sys.modules mock injection above) - - -@mock.patch("cotdata.providers.norgate.store.read_prices") -@mock.patch("norgatedata.database_symbols", create=True) -@mock.patch("cotdata.providers.norgate.store.write_prices") -@mock.patch("norgatedata.price_timeseries", create=True) -def test_norgate_update_fetches_both_adjustments(mock_price_ts, mock_write_prices, mock_db_symbols, mock_read_prices): - """Verify that update() fetches both the backadj and unadj series for a symbol.""" - mock_db_symbols.return_value = [] - mock_read_prices.return_value = pd.DataFrame() - - # Setup mock returns - mock_df = pd.DataFrame({ - "Open": [100.0], "High": [101.0], "Low": [99.0], "Close": [100.5], - "Volume": [1000], "Open Interest": [5000], "Delivery Month": [202609] - }, index=pd.DatetimeIndex(["2026-07-01"])) - - mock_price_ts.return_value = mock_df - - # Mock all_symbols to just return a dummy registry entry for "ES" - mock_symbol = mock.Mock() - mock_symbol.internal = "ES" - - # We must patch REGISTRY and all_symbols so it uses our mock - with mock.patch("cotdata.providers.norgate.all_symbols", return_value=[mock_symbol]), \ - mock.patch.dict("cotdata.providers.norgate.REGISTRY", {"ES": mock.Mock(norgate="&ES")}): - - # Run update - norgate.update(symbols=["ES"]) - - # Verify norgatedata API was called twice with correct raw symbols - assert mock_price_ts.call_count == 2 - calls = mock_price_ts.call_args_list - # Call 1: backadj ("&ES_CCB") - assert calls[0][0][0] == "&ES_CCB" - # Call 2: unadj ("&ES") - assert calls[1][0][0] == "&ES" - - # Verify store.write_prices was called twice with correct adjustment flags - assert mock_write_prices.call_count == 2 - write_calls = mock_write_prices.call_args_list - # Call 1: store.write_prices("ES", "backadj", out_backadj, source="norgate") - assert write_calls[0][0][0] == "ES" - assert write_calls[0][0][1] == "backadj" - # Call 2: store.write_prices("ES", "unadj", out_unadj, source="norgate") - assert write_calls[1][0][0] == "ES" - assert write_calls[1][0][1] == "unadj" - -@mock.patch("cotdata.providers.norgate.store.write_prices") -@mock.patch("cotdata.providers.norgate.store.read_prices") -@mock.patch("norgatedata.database_symbols", create=True) -@mock.patch("norgatedata.price_timeseries", create=True) -def test_volume_reconstruction(mock_price_ts, mock_db_symbols, mock_read_prices, mock_write_prices): - """Verify that _reconstruct_volume correctly appends additive columns without modifying default Volume.""" - - # Mock continuous dataframe - mock_continuous = pd.DataFrame({ - "Open": [100.0], "High": [101.0], "Low": [99.0], "Close": [100.5], - "Volume": [1000], "Open Interest": [5000], "Delivery Month": [202609] - }, index=pd.DatetimeIndex(["2026-07-01"])) - - # Mock individual contracts - mock_indiv_H = pd.DataFrame({ - "Date": [pd.Timestamp("2026-07-01")], - "Volume": [600], - "Open Interest": [3000] - }) - mock_indiv_M = pd.DataFrame({ - "Date": [pd.Timestamp("2026-07-01")], - "Volume": [400], - "Open Interest": [2000] - }) - - def mock_ts_side_effect(sym, **kwargs): - if sym.endswith("CCB") or "-" not in sym: - return mock_continuous.copy() - if sym == "ES-2026H": - return mock_indiv_H.copy() - if sym == "ES-2026M": - return mock_indiv_M.copy() - return pd.DataFrame() - - mock_price_ts.side_effect = mock_ts_side_effect - mock_db_symbols.return_value = ["ES-2026H", "ES-2026M", "ES-2025Z"] - - # Mock existing prices to trigger full backfill - mock_read_prices.return_value = pd.DataFrame() - - mock_symbol = mock.Mock() - mock_symbol.internal = "ES" - - with mock.patch("cotdata.providers.norgate.all_symbols", return_value=[mock_symbol]), \ - mock.patch.dict("cotdata.providers.norgate.REGISTRY", {"ES": mock.Mock(norgate="&ES")}): - - norgate.update(symbols=["ES"]) - - # Verify the written dataframe has the additive columns and default Volume is untouched - write_call = mock_write_prices.call_args_list[0] - written_df = write_call[0][2] - - assert "FirstVolume" in written_df.columns - assert "SecondVolume" in written_df.columns - assert "Volume_Reconstructed" in written_df.columns - assert "Volume_Source" in written_df.columns - - # Default Volume should be UNTOUCHED (1000) - assert written_df["Volume"].iloc[0] == 1000 - - # FirstVolume (ES-2026H) + SecondVolume (ES-2026M) = 1000 - assert written_df["FirstVolume"].iloc[0] == 600 - assert written_df["SecondVolume"].iloc[0] == 400 - assert written_df["Volume_Reconstructed"].iloc[0] == 1000 - assert written_df["Volume_Source"].iloc[0] == "reconstructed" - assert written_df["FirstContract"].iloc[0] == "ES-2026H" - assert written_df["SecondContract"].iloc[0] == "ES-2026M" - - -@mock.patch("cotdata.providers.norgate.store.write_prices") -@mock.patch("cotdata.providers.norgate.store.read_prices") -@mock.patch("norgatedata.database_symbols", create=True) -@mock.patch("norgatedata.price_timeseries", create=True) -def test_reconstruction_picks_by_volume_not_expiry(mock_price_ts, mock_db_symbols, mock_read_prices, mock_write_prices): - """First/Second must be the two HIGHEST-VOLUME contracts, not the two nearest by - expiry. Models the GC/SI case: the nearest serial month is near-empty while a - later contract is dominant. An expiry-order pick would name the empty serial as - 'First' and understate volume; volume-rank must name the dominant contract.""" - mock_continuous = pd.DataFrame({ - "Open": [100.0], "High": [101.0], "Low": [99.0], "Close": [100.5], - "Volume": [1000], "Open Interest": [5000], "Delivery Month": [202606], - }, index=pd.DatetimeIndex(["2026-07-01"])) - - # ES-2026H (March) = nearest by expiry but near-empty; ES-2026M (June) = dominant. - near_empty = pd.DataFrame({"Date": [pd.Timestamp("2026-07-01")], "Volume": [50]}) - dominant = pd.DataFrame({"Date": [pd.Timestamp("2026-07-01")], "Volume": [900]}) - - def mock_ts_side_effect(sym, **kwargs): - if sym.endswith("CCB") or "-" not in sym: - return mock_continuous.copy() - if sym == "ES-2026H": - return near_empty.copy() - if sym == "ES-2026M": - return dominant.copy() - return pd.DataFrame() - - mock_price_ts.side_effect = mock_ts_side_effect - mock_db_symbols.return_value = ["ES-2026H", "ES-2026M"] - mock_read_prices.return_value = pd.DataFrame() - - mock_symbol = mock.Mock() - mock_symbol.internal = "ES" - with mock.patch("cotdata.providers.norgate.all_symbols", return_value=[mock_symbol]), \ - mock.patch.dict("cotdata.providers.norgate.REGISTRY", {"ES": mock.Mock(norgate="&ES")}): - norgate.update(symbols=["ES"]) - written_df = mock_write_prices.call_args_list[0][0][2] - - # Dominant (June, 900) is First even though March expires sooner. - assert written_df["FirstContract"].iloc[0] == "ES-2026M" - assert written_df["FirstVolume"].iloc[0] == 900 - assert written_df["SecondContract"].iloc[0] == "ES-2026H" - assert written_df["SecondVolume"].iloc[0] == 50 - assert written_df["Volume_Reconstructed"].iloc[0] == 950 - - -@mock.patch("cotdata.providers.norgate.store.write_prices") -@mock.patch("cotdata.providers.norgate.store.read_prices") -@mock.patch("norgatedata.database_symbols", create=True) -@mock.patch("norgatedata.price_timeseries", create=True) -def test_volume_reconstruction_incremental(mock_price_ts, mock_db_symbols, mock_read_prices, mock_write_prices): - """Verify that _reconstruct_volume preserves old Volume_Source during an incremental run.""" - - # Existing df has an old date with a "raw" fallback, and a slightly newer one with "reconstructed" - mock_existing = pd.DataFrame({ - "Volume": [500, 800], - "Volume_Reconstructed": [500, 800], - "FirstVolume": [np.nan, 500], - "SecondVolume": [np.nan, 300], - "FirstContract": ["", "ES-2026H"], - "SecondContract": ["", "ES-2026M"], - "Volume_Source": ["raw", "reconstructed"] - }, index=pd.DatetimeIndex(["2020-01-01", "2026-06-01"])) - - mock_read_prices.return_value = mock_existing.copy() - - # New continuous dataframe has the old dates + a new date - mock_continuous = pd.DataFrame({ - "Open": [10, 10, 10], "High": [10, 10, 10], "Low": [10, 10, 10], "Close": [10, 10, 10], - "Volume": [500, 800, 1000], "Open Interest": [0, 0, 0], "Delivery Month": [0, 0, 0] - }, index=pd.DatetimeIndex(["2020-01-01", "2026-06-01", "2026-07-01"])) - - # The new date gets fetched. The trailing 60 days from 2026-06-01 is 2026-04-02. - # We will just return some mock individual contracts. - mock_indiv_U = pd.DataFrame({ - "Date": [pd.Timestamp("2026-07-01")], - "Volume": [600], "Open Interest": [0] - }) - mock_indiv_Z = pd.DataFrame({ - "Date": [pd.Timestamp("2026-07-01")], - "Volume": [400], "Open Interest": [0] - }) - - def mock_ts_side_effect(sym, **kwargs): - if sym.endswith("CCB") or "-" not in sym: - return mock_continuous.copy() - if sym == "ES-2026U": - return mock_indiv_U.copy() - if sym == "ES-2026Z": - return mock_indiv_Z.copy() - return pd.DataFrame() - - mock_price_ts.side_effect = mock_ts_side_effect - mock_db_symbols.return_value = ["ES-2026U", "ES-2026Z"] - - mock_symbol = mock.Mock() - mock_symbol.internal = "ES" - - with mock.patch("cotdata.providers.norgate.all_symbols", return_value=[mock_symbol]), \ - mock.patch.dict("cotdata.providers.norgate.REGISTRY", {"ES": mock.Mock(norgate="&ES")}): - - norgate.update(symbols=["ES"]) - - write_call = mock_write_prices.call_args_list[0] - written_df = write_call[0][2] - - # Verify the 2020-01-01 row is still "raw" - assert written_df.loc["2020-01-01", "Volume_Source"] == "raw" - assert written_df.loc["2020-01-01", "Volume_Reconstructed"] == 500 - - # Verify the 2026-06-01 row is still "reconstructed" - assert written_df.loc["2026-06-01", "Volume_Source"] == "reconstructed" - - # Verify the newly fetched 2026-07-01 row is computed correctly - assert written_df.loc["2026-07-01", "Volume_Source"] == "reconstructed" - assert written_df.loc["2026-07-01", "Volume_Reconstructed"] == 1000 - assert written_df.loc["2026-07-01", "FirstContract"] == "ES-2026U" - - -@mock.patch("cotdata.providers.norgate.store.write_prices") -@mock.patch("cotdata.providers.norgate.store.read_prices") -@mock.patch("norgatedata.database_symbols", create=True) -@mock.patch("norgatedata.price_timeseries", create=True) -def test_full_rebuild_bypasses_incremental_window(mock_price_ts, mock_db_symbols, mock_read_prices, mock_write_prices): - """update(full=True) must recompute from epoch, ignoring the trailing-60-day - window — even when the store already carries recent Volume_Reconstructed. The - individual-contract fetch should be issued with start_date=1970-01-01.""" - mock_existing = pd.DataFrame({ - "Volume": [800], - "Volume_Reconstructed": [800], - "FirstVolume": [500], "SecondVolume": [300], - "FirstContract": ["ES-2026H"], "SecondContract": ["ES-2026M"], - "Volume_Source": ["reconstructed"], - }, index=pd.DatetimeIndex(["2026-06-01"])) - mock_read_prices.return_value = mock_existing.copy() - - mock_continuous = pd.DataFrame({ - "Open": [10, 10], "High": [10, 10], "Low": [10, 10], "Close": [10, 10], - "Volume": [800, 1000], "Open Interest": [0, 0], "Delivery Month": [0, 0], - }, index=pd.DatetimeIndex(["2026-06-01", "2026-07-01"])) - mock_indiv = pd.DataFrame({"Date": [pd.Timestamp("2026-07-01")], "Volume": [1000]}) - - def mock_ts_side_effect(sym, **kwargs): - if sym.endswith("CCB") or "-" not in sym: - return mock_continuous.copy() - if sym == "ES-2026U": - return mock_indiv.copy() - return pd.DataFrame() - - mock_price_ts.side_effect = mock_ts_side_effect - mock_db_symbols.return_value = ["ES-2026U"] - - mock_symbol = mock.Mock() - mock_symbol.internal = "ES" - with mock.patch("cotdata.providers.norgate.all_symbols", return_value=[mock_symbol]), \ - mock.patch.dict("cotdata.providers.norgate.REGISTRY", {"ES": mock.Mock(norgate="&ES")}): - norgate.update(symbols=["ES"], full=True) - - # Every individual-contract fetch (sym containing '-') must start from epoch. - indiv_starts = [c.kwargs.get("start_date") for c in mock_price_ts.call_args_list - if "-" in c.args[0]] - assert indiv_starts, "expected at least one individual-contract fetch" - assert all(s == "1970-01-01" for s in indiv_starts), indiv_starts - - -def test_scoped_update_metadata_upserts_preserving_others(tmp_path, monkeypatch): - """`update_metadata(symbols=[...])` must UPSERT by Symbol into the existing - contract_specs — the data-loss regression where a 5-symbol run replaced the - whole 42-market table. Untouched markets survive; requested ones are refreshed. - Exercises the real store round-trip through a tmp COTDATA_STORE.""" - monkeypatch.setenv("COTDATA_STORE", str(tmp_path)) - from cotdata import store - - # Pre-existing full table (stand-in for the 42 markets already on disk) - store.write_metadata( - pd.DataFrame({"Symbol": ["ES", "NQ", "DC"], "Tick Size": [0.25, 0.25, 0.01]}), - source="seed", - ) - - def fake_meta(sym): - return {"Symbol": sym, "Tick Size": 99.0} # sentinel refreshed value - - with mock.patch("cotdata.providers.norgate.get_symbol_metadata", side_effect=fake_meta): - norgate.update_metadata(symbols=["DC"]) - - df = store.read_metadata().set_index("Symbol") - assert set(df.index) == {"ES", "NQ", "DC"} # ES/NQ preserved — not dropped - assert df.loc["DC", "Tick Size"] == 99.0 # DC refreshed - assert df.loc["ES", "Tick Size"] == 0.25 # untouched market unchanged - - -def test_full_update_metadata_replaces_table(tmp_path, monkeypatch): - """`update_metadata()` with no symbols regenerates the whole registry and - REPLACES the table (drops symbols no longer produced).""" - monkeypatch.setenv("COTDATA_STORE", str(tmp_path)) - from cotdata import store - - store.write_metadata( - pd.DataFrame({"Symbol": ["OLD"], "Tick Size": [1.0]}), source="seed", - ) - - sym_a = mock.Mock(internal="ES") - sym_b = mock.Mock(internal="NQ") - with mock.patch("cotdata.providers.norgate.all_symbols", return_value=[sym_a, sym_b]), \ - mock.patch("cotdata.providers.norgate.get_symbol_metadata", - side_effect=lambda s: {"Symbol": s, "Tick Size": 1.0}): - norgate.update_metadata() # no symbols → full replace - - assert set(store.read_metadata()["Symbol"]) == {"ES", "NQ"} # OLD gone - - -def test_update_aborts_fast_when_ndu_unreachable(monkeypatch): - """When NDU is down, update() must raise BEFORE any fetch — not fall into - norgatedata's 10x-retry + bare sys.exit() (which exits 0 and defeats scheduler - retry). norgatedata.status() returning False is the trip wire.""" - monkeypatch.setattr(mock_norgatedata, "status", mock.Mock(return_value=False)) - with pytest.raises(RuntimeError, match="Norgate Data service is not reachable"): - norgate.update(symbols=["ES"]) - - -def test_update_metadata_aborts_fast_when_ndu_unreachable(monkeypatch): - monkeypatch.setattr(mock_norgatedata, "status", mock.Mock(return_value=False)) - with pytest.raises(RuntimeError, match="Norgate Data service is not reachable"): - norgate.update_metadata(symbols=["ES"]) - - -def test_metadata_skips_all_null_spec_rows(tmp_path, monkeypatch): - """A COVERED symbol whose specs all come back None (a transient Norgate failure, - not the MME/MFS no-coverage case) must be skipped — never written as a null row, - and on a scoped upsert never used to overwrite good existing specs.""" - monkeypatch.setenv("COTDATA_STORE", str(tmp_path)) - from cotdata import store - - sym_es = mock.Mock(internal="ES", norgate="&ES") - sym_nq = mock.Mock(internal="NQ", norgate="&NQ") - - def fake_meta(sym): - base = {"Symbol": sym, "Norgate_Symbol": f"&{sym}_CCB"} - if sym == "NQ": # all specs empty → junk - return {**base, **{k: None for k in norgate._SPEC_FIELDS}} - return {**base, **{k: None for k in norgate._SPEC_FIELDS}, "Tick Size": 0.25} - - with mock.patch("cotdata.providers.norgate.all_symbols", return_value=[sym_es, sym_nq]), \ - mock.patch.dict("cotdata.providers.norgate.REGISTRY", {"ES": sym_es, "NQ": sym_nq}), \ - mock.patch("cotdata.providers.norgate.get_symbol_metadata", side_effect=fake_meta): - norgate.update_metadata() # full run - - assert set(store.read_metadata()["Symbol"]) == {"ES"} # NQ null row skipped - - -def test_metadata_skips_symbols_without_norgate_coverage(tmp_path, monkeypatch): - """Yahoo-only markets (registry norgate=None, e.g. MME/MFS) must be skipped by - the Norgate metadata producer — never fetched, never written as null rows. The - regression: `&MME_CCB not found` spam + all-null spec rows in contract_specs.""" - monkeypatch.setenv("COTDATA_STORE", str(tmp_path)) - from cotdata import store - - sym_es = mock.Mock(internal="ES", norgate="&ES") - sym_mme = mock.Mock(internal="MME", norgate=None) # no Norgate coverage - called = [] - - def fake_meta(sym): - called.append(sym) - return {"Symbol": sym, "Tick Size": 1.0} - - with mock.patch("cotdata.providers.norgate.all_symbols", - return_value=[sym_es, sym_mme]), \ - mock.patch.dict("cotdata.providers.norgate.REGISTRY", - {"ES": sym_es, "MME": sym_mme}), \ - mock.patch("cotdata.providers.norgate.get_symbol_metadata", - side_effect=fake_meta): - norgate.update_metadata() # full run - - assert called == ["ES"] # MME never fetched - assert set(store.read_metadata()["Symbol"]) == {"ES"} # no null MME row - - -def test_covered_filter_drops_none_norgate(): - """Unit: _norgate_covered keeps only symbols whose registry norgate is truthy.""" - with mock.patch.dict("cotdata.providers.norgate.REGISTRY", - {"ES": mock.Mock(norgate="&ES"), - "MME": mock.Mock(norgate=None)}): - assert norgate._norgate_covered(["ES", "MME"]) == ["ES"] - - - - -def test_finals_ready_pure_logic(): - from cotdata.providers.norgate import _finals_ready - now = _dt.datetime(2026, 7, 15, 21, 30) # 9:30pm local - after = _dt.datetime(2026, 7, 15, 20, 56) # updated after 20:55 cutoff - before = _dt.datetime(2026, 7, 15, 20, 40) # updated before cutoff - # both DBs refreshed after cutoff -> ready - ok, _ = _finals_ready({"Futures": after, "Continuous Futures": after}, "20:55", now) - assert ok is True - # one DB still on pre-cutoff (interim) data -> not ready - ng, _ = _finals_ready({"Futures": after, "Continuous Futures": before}, "20:55", now) - assert ng is False - # missing update time -> not ready - nn, _ = _finals_ready({"Futures": None, "Continuous Futures": after}, "20:55", now) - assert nn is False - - -def test_finals_ready_handles_tz_aware_times(): - """norgatedata returns tz-aware datetimes (e.g. -04:00); comparing them against - a naive cutoff must not raise, and must evaluate by local wall-clock.""" - import datetime as d - - from cotdata.providers.norgate import _finals_ready - et = d.timezone(d.timedelta(hours=-4)) - now = d.datetime(2026, 7, 15, 21, 30) # 9:30pm naive local - after = d.datetime(2026, 7, 15, 20, 56, tzinfo=et) # aware, after 20:55 - before = d.datetime(2026, 7, 15, 6, 12, tzinfo=et) # aware, morning update - ok, _ = _finals_ready({"Futures": after, "Continuous Futures": after}, "20:55", now) - assert ok is True - ng, _ = _finals_ready({"Futures": after, "Continuous Futures": before}, "20:55", now) - assert ng is False - - -def test_finals_ready_by_date_pure_logic(): - """Data-driven gate: ready iff Norgate has a newer settled bar than the store.""" - import datetime as d - - from cotdata.providers.norgate import _finals_ready_by_date - mon, tue = d.date(2026, 7, 27), d.date(2026, 7, 28) - # a newer settled session is available -> ready - ok, det = _finals_ready_by_date(tue, mon) - assert ok is True and det["norgate_last"] == "2026-07-28" - # store already holds the latest -> nothing new, not ready - assert _finals_ready_by_date(mon, mon)[0] is False - # Norgate somehow behind the store -> not ready - assert _finals_ready_by_date(mon, tue)[0] is False - # empty store -> ready on any Norgate bar - assert _finals_ready_by_date(mon, None)[0] is True - # Norgate offers nothing -> defer - assert _finals_ready_by_date(None, mon)[0] is False - - -def test_finals_ready_by_date_normalizes_datetimes(): - """datetimes collapse to their date: same calendar day is not 'newer'.""" - import datetime as d - - from cotdata.providers.norgate import _finals_ready_by_date - same_day = _finals_ready_by_date(d.datetime(2026, 7, 27, 23, 0), d.datetime(2026, 7, 27, 8, 0)) - assert same_day[0] is False - next_day = _finals_ready_by_date(d.datetime(2026, 7, 28, 9, 0), d.datetime(2026, 7, 27, 20, 0)) - assert next_day[0] is True - - -def test_finals_ready_quorum(): - """Ready only when EVERY reference symbol has a newer settled bar than the store.""" - import datetime as d - - from cotdata.providers.norgate import _finals_ready_quorum - mon, tue = d.date(2026, 7, 27), d.date(2026, 7, 28) - ok, det = _finals_ready_quorum( - {"ES": tue, "CL": tue, "ZC": tue}, {"ES": mon, "CL": mon, "ZC": mon}) - assert ok is True and det["mode"] == "data" - # one reference still lagging (Norgate hasn't published it) -> defer, wait for all - ng, _ = _finals_ready_quorum( - {"ES": tue, "CL": mon, "ZC": tue}, {"ES": mon, "CL": mon, "ZC": mon}) - assert ng is False - # nothing new anywhere -> not ready - assert _finals_ready_quorum({"ES": mon, "CL": mon}, {"ES": mon, "CL": mon})[0] is False - - -def test_finals_ready_gathers_and_delegates(monkeypatch): - """finals_ready reads norgate+store dates per ref symbol and delegates to the quorum; - the legacy `cutoff` arg is accepted but ignored.""" - import datetime as d - - from cotdata.providers import norgate - monkeypatch.setattr(norgate, "_require_norgate_service", lambda: None) - ng = {s: d.date(2026, 7, 28) for s in norgate._FINALS_REF_SYMBOLS} - st = {s: d.date(2026, 7, 27) for s in norgate._FINALS_REF_SYMBOLS} - monkeypatch.setattr(norgate, "_norgate_last_bar_date", lambda s: ng[s]) - monkeypatch.setattr(norgate, "_store_last_bar_date", lambda s: st[s]) - ready, detail = norgate.finals_ready() - assert ready is True - assert set(detail["per_symbol"]) == set(norgate._FINALS_REF_SYMBOLS) - assert norgate.finals_ready("20:55")[0] is True # cutoff ignored - - # a lagging reference (Norgate hasn't advanced it) keeps the whole gate closed - monkeypatch.setattr(norgate, "_norgate_last_bar_date", - lambda s: d.date(2026, 7, 27) if s == norgate._FINALS_REF_SYMBOLS[0] else ng[s]) - assert norgate.finals_ready()[0] is False diff --git a/tests/test_propadj.py b/tests/test_propadj.py deleted file mode 100644 index 38b120f..0000000 --- a/tests/test_propadj.py +++ /dev/null @@ -1,142 +0,0 @@ -"""Proportional (ratio) back-adjustment derived on read — get_prices(..., 'propadj'). - -Models the DC / Class III Milk failure mode: a low-priced contract whose ADDITIVE -back-adjustment (Norgate _CCB) accumulates roll gaps below zero, which breaks -price-based stops and R-multiples. propadj must recover a strictly-positive series -that preserves percentage returns and stays sign-identical to backadj at rolls. -""" -import numpy as np -import pandas as pd -import pytest - - -@pytest.fixture() -def store_env(tmp_path, monkeypatch): - monkeypatch.setenv("COTDATA_STORE", str(tmp_path)) - return tmp_path - - -def _milk_like(with_delivery_month=True): - """Three contract segments (two rolls) of a ~$2–4 low-priced contract. - - Offsets O = B − U are piecewise-constant (−3.0, −1.5, 0), so the additive - close B goes negative in the oldest segment while the unadjusted close U is - always positive — exactly DC's situation, in miniature. - """ - idx = pd.date_range("2020-01-01", periods=9, freq="D", name="Date") - u_close = np.array([2.0, 2.1, 2.2, 3.0, 3.1, 3.2, 4.0, 4.1, 4.2]) - offset = np.array([-3.0] * 3 + [-1.5] * 3 + [0.0] * 3) - dm = ["202003"] * 3 + ["202006"] * 3 + ["202009"] * 3 - - U = pd.DataFrame({ - "Open": u_close - 0.05, "High": u_close + 0.10, - "Low": u_close - 0.10, "Close": u_close, - "Volume": [10] * 9, "Open Interest": [100] * 9, - }, index=idx) - B = U.copy() - for c in ("Open", "High", "Low", "Close"): - B[c] = U[c] + offset - if with_delivery_month: - U["Delivery Month"] = dm - B["Delivery Month"] = dm - return U, B - - -def _write(sym="DC", **kw): - from cotdata import store - U, B = _milk_like(**kw) - store.write_prices(sym, "unadj", U, source="test") - store.write_prices(sym, "backadj", B, source="test") - return U, B - - -def test_backadj_goes_negative_but_propadj_is_strictly_positive(store_env): - from cotdata import get_prices - _write() - assert (get_prices("DC", "backadj")["Close"] <= 0).any() # the problem - p = get_prices("DC", "propadj") - assert (p[["Open", "High", "Low", "Close"]] > 0).all().all() # the fix - - -def test_anchored_to_actual_recent_price(store_env): - """Most-recent segment keeps actual (unadjusted) prices — factor == 1.""" - from cotdata import get_prices - U, _ = _write() - p = get_prices("DC", "propadj") - assert p["Close"].iloc[-3:].tolist() == pytest.approx(U["Close"].iloc[-3:].tolist()) - - -def test_preserves_within_segment_pct_returns(store_env): - from cotdata import get_prices - U, _ = _write() - p = get_prices("DC", "propadj") - dm = p["Delivery Month"] - non_roll = ~(dm.ne(dm.shift()) & dm.shift().notna()) - err = (p["Close"].pct_change() - U["Close"].pct_change())[non_roll].abs() - assert err.max() < 1e-12 - - -def test_sign_identical_to_backadj_including_rolls(store_env): - """Ratio- and additive-adjustment remove the same roll gaps, so every daily - move — including across rolls — must agree in direction.""" - from cotdata import get_prices - _write() - p = get_prices("DC", "propadj")["Close"].diff() - b = get_prices("DC", "backadj")["Close"].diff() - both = p.notna() & b.notna() & (b.abs() > 1e-12) - assert (np.sign(p[both]) == np.sign(b[both])).all() - - -def test_hand_computed_factors(store_env): - """Regression guard on the exact ratio construction. - - spread(roll) = O[r−1] − O[r]; k = (U[r−1] + spread) / U[r−1]; segment factor = - product of k for rolls at/after it, most-recent segment anchored to 1. - roll@day3: spread = −3.0−(−1.5) = −1.5, U_prev = 2.2 → k1 = 0.7/2.2 - roll@day6: spread = −1.5−0 = −1.5, U_prev = 3.2 → k2 = 1.7/3.2 - factor[seg2]=1, factor[seg1]=k2, factor[seg0]=k1*k2 - """ - from cotdata import get_prices - _write() - c = get_prices("DC", "propadj")["Close"] - k1, k2 = 0.7 / 2.2, 1.7 / 3.2 - assert c.iloc[0] == pytest.approx(2.0 * k1 * k2) # oldest segment - assert c.iloc[3] == pytest.approx(3.0 * k2) # middle segment - assert c.iloc[6] == pytest.approx(4.0) # anchor segment, factor 1 - - -def test_ohlc_ordering_preserved(store_env): - from cotdata import get_prices - _write() - p = get_prices("DC", "propadj") - assert (p["High"] >= p["Close"]).all() and (p["Close"] >= p["Low"]).all() - assert (p["High"] >= p["Open"]).all() and (p["Open"] >= p["Low"]).all() - - -def test_falls_back_to_offset_jumps_without_delivery_month(store_env): - """Rolls are still detected from offset steps when Delivery Month is absent.""" - from cotdata import get_prices - _write(with_delivery_month=False) - p = get_prices("DC", "propadj") - assert (p["Close"] > 0).all() - assert p["Close"].iloc[0] == pytest.approx(2.0 * (0.7 / 2.2) * (1.7 / 3.2)) - - -def test_empty_when_either_series_missing(store_env): - from cotdata import get_prices, store - U, _ = _milk_like() - store.write_prices("DC", "unadj", U, source="test") # no backadj written - assert get_prices("DC", "propadj").empty - - -def test_reconstructed_volume_view_still_works_on_propadj(store_env): - """The volume view composes with the derived adjustment.""" - from cotdata import get_prices, store - U, B = _milk_like() - U["Volume_Reconstructed"] = [15, 14, 10, 18, 20, 22, 25, 24, 26] - U["Volume_Source"] = ["reconstructed"] * 9 - store.write_prices("DC", "unadj", U, source="test") - store.write_prices("DC", "backadj", B, source="test") - p = get_prices("DC", "propadj", volume="reconstructed") - assert p["Volume"].tolist() == [15, 14, 10, 18, 20, 22, 25, 24, 26] - assert "Volume_Source" in p.columns diff --git a/tests/test_store.py b/tests/test_store.py index c81743b..7e9a0dc 100644 --- a/tests/test_store.py +++ b/tests/test_store.py @@ -21,12 +21,18 @@ def _sample(): def test_prices_roundtrip_and_manifest(store_env): - from cotdata import get_prices, load_manifest, store + """The store-level price pair, which the databento producer still writes through. + + The CONSUMER bar API (get_prices/roll_dates/propadj, the volume views) left with + ADR-0007 and is tested in marketdata now — see test_consumer_bar_api_is_gone. + """ + from cotdata import load_manifest, store store.write_prices("ES", "backadj", _sample(), source="test") - df = get_prices("ES", "backadj") + df = store.read_prices("ES", "backadj") assert list(df.columns)[:6] == ["Open", "High", "Low", "Close", "Volume", "Open Interest"] - assert df.index.name == "Date" and len(df) == 5 + assert len(df) == 5 + assert store.read_prices("ZZ", "backadj").empty # absent symbol -> empty m = load_manifest() assert m["prices"]["ES_backadj"]["n_rows"] == 5 @@ -63,106 +69,37 @@ def test_reconcile_noop_when_clean(store_env): assert store.reconcile_manifest() == {} # nothing to prune -def test_roll_dates_from_delivery_month(store_env): - from cotdata import roll_dates, store - store.write_prices("ES", "backadj", _sample(), source="test") - rolls = roll_dates("ES", "backadj") - # first bar + the delivery-month change on day 4 - assert len(rolls) == 2 - assert pd.Timestamp("2020-01-04") in rolls - - -def test_missing_symbol_returns_empty(store_env): - from cotdata import get_prices - assert get_prices("ZZ", "backadj").empty - - -def _sample_reconstructed(): - """Sample carrying the v2 reconstruction columns, with one 'raw' fallback row - (row 2) where no individual contracts were available.""" - df = _sample() - df["Volume_Reconstructed"] = [15, 14, 10, 18, 20] # row idx 2 == front-month (raw) - df["Volume_Source"] = ["reconstructed", "reconstructed", "raw", - "reconstructed", "reconstructed"] - return df - - -def test_default_volume_view_is_byte_identical(store_env): - """volume='front' (default) must not change the pre-v2 output shape.""" - from cotdata import get_prices, store - store.write_prices("ES", "backadj", _sample_reconstructed(), source="test") - - df = get_prices("ES", "backadj") # default volume='front' - assert list(df.columns) == ["Open", "High", "Low", "Close", "Volume", - "Open Interest", "Delivery Month"] - assert "Volume_Source" not in df.columns - assert "Volume_Reconstructed" not in df.columns - assert df["Volume"].tolist() == [10] * 5 # untouched front-month - - -def test_reconstructed_volume_view(store_env): - """volume='reconstructed' swaps Volume in, keeps per-row raw fallback, and - surfaces Volume_Source for audit.""" - from cotdata import get_prices, store - store.write_prices("ES", "backadj", _sample_reconstructed(), source="test") +def test_consumer_bar_api_is_gone(store_env): + """ADR-0007 §7.5: this package answers positioning questions, not price ones. - df = get_prices("ES", "backadj", volume="reconstructed") - assert "Volume_Source" in df.columns - assert "Volume_Reconstructed" not in df.columns # internal — not leaked - # reconstructed values flow into Volume; the 'raw' row keeps front-month (10) - assert df["Volume"].tolist() == [15, 14, 10, 18, 20] - assert df["Volume_Source"].tolist() == ["reconstructed", "reconstructed", - "raw", "reconstructed", "reconstructed"] + Asserted rather than assumed. A re-export costs one line to add back, and a + consumer that finds `cotdata.get_prices` importable again will use it — landing + on a store the nightly job no longer fills, which reads as stale data rather + than as a wrong import. Bars come from `marketdata.get_bars`. + """ + import importlib + import cotdata -def test_reconstructed_view_falls_back_on_pre_v2_store(store_env): - """A store written before reconstruction existed → reconstructed view returns - front-month volume labelled 'raw', never NaN.""" - from cotdata import get_prices, store - store.write_prices("ES", "backadj", _sample(), source="test") # no recon cols + for name in ("get_prices", "roll_dates"): + assert not hasattr(cotdata, name) + assert name not in cotdata.__all__ + for mod in ("cotdata.prices", "cotdata.providers.norgate", "cotdata.providers.yfinance"): + with pytest.raises(ModuleNotFoundError): + importlib.import_module(mod) - df = get_prices("ES", "backadj", volume="reconstructed") - assert df["Volume"].tolist() == [10] * 5 - assert (df["Volume_Source"] == "raw").all() +def test_contract_specs_are_no_longer_written_here(store_env): + """Contract specs moved to marketdata with the bars they describe (§7.2). -def test_invalid_volume_arg_raises(store_env): - from cotdata import get_prices, store - store.write_prices("ES", "backadj", _sample(), source="test") - with pytest.raises(ValueError): - get_prices("ES", "backadj", volume="bogus") - - -def _specs(symbols, tick=1.0): - """Minimal contract_specs frame (Symbol-keyed, RangeIndex — as norgate writes).""" - return pd.DataFrame({"Symbol": symbols, "Tick Size": [tick] * len(symbols)}) - - -def test_upsert_metadata_preserves_unlisted_symbols(store_env): - """A scoped upsert must keep rows for symbols not in the incoming frame — the - data-loss regression: writing 5 symbols must not drop the other 42.""" + The domain stays DECLARED so pre-ADR-0007 stores still migrate and reconcile + their `metadata` entries instead of stranding them — but there is no writer. + """ from cotdata import store - store.write_metadata(_specs(["ES", "NQ", "DC", "ZO", "KE"]), source="test") - - # Scoped refresh of only DC + a brand-new symbol - store.upsert_metadata(_specs(["DC", "EMD"], tick=9.9), source="norgate") - - df = store.read_metadata() - assert set(df["Symbol"]) == {"ES", "NQ", "DC", "ZO", "KE", "EMD"} # none dropped - # DC replaced with the new value; untouched markets keep their original - assert df.set_index("Symbol").loc["DC", "Tick Size"] == 9.9 - assert df.set_index("Symbol").loc["ES", "Tick Size"] == 1.0 - assert df.set_index("Symbol").loc["EMD", "Tick Size"] == 9.9 # new row added - # manifest reflects the union, not just the 2 upserted rows - assert store.load_manifest()["metadata"]["contract_specs"]["n_rows"] == 6 - - -def test_upsert_metadata_on_empty_store_writes_all(store_env): - """Upsert against an empty store behaves like a plain write.""" - from cotdata import store - store.upsert_metadata(_specs(["ES", "NQ"]), source="norgate") - assert set(store.read_metadata()["Symbol"]) == {"ES", "NQ"} + assert store.half_for("metadata") == "prices" # still mapped, for old stores + for name in ("write_metadata", "upsert_metadata", "read_metadata"): + assert not hasattr(store, name) def test_schema_version_and_require_schema(store_env): diff --git a/tests/test_validate_databento.py b/tests/test_validate_databento.py index 5a88778..6e01c3c 100644 --- a/tests/test_validate_databento.py +++ b/tests/test_validate_databento.py @@ -98,10 +98,16 @@ def test_roll_dates_from_delivery_month(): assert m["rolls_norgate"] == 1 and m["rolls_common"] == 1 -def test_read_backadj_roundtrip(tmp_path): - prices = tmp_path / "prices" - prices.mkdir() - _frame(_NG_CLOSE).to_parquet(prices / "ES_backadj.parquet") +@pytest.mark.parametrize("layout", ["prices", "bars/futures/norgate"]) +def test_read_backadj_roundtrip(tmp_path, layout): + """Both store layouts, because this harness now spans two packages: ADR-0007 + moved the Norgate side into marketdata (`bars///`) while the + databento side still writes cotdata's flat `prices/`. Reading only one would + silently report the Norgate store as absent and skip every symbol — a green + run that compared nothing.""" + d = tmp_path / layout + d.mkdir(parents=True) + _frame(_NG_CLOSE).to_parquet(d / "ES_backadj.parquet") got = val.read_backadj(str(tmp_path), "ES") assert not got.empty and got.index.name == "Date" and got.index.tz is None assert val.read_backadj(str(tmp_path), "NOPE").empty diff --git a/tests/test_yfinance_provider.py b/tests/test_yfinance_provider.py deleted file mode 100644 index a67f870..0000000 --- a/tests/test_yfinance_provider.py +++ /dev/null @@ -1,61 +0,0 @@ -"""Hermetic tests for the yfinance price provider — no network, no store writes, and -no real yfinance dependency (it's an optional [yahoo] extra, absent in CI). - -Injects a stub `yfinance` module into sys.modules (the provider does `import yfinance` -lazily inside _fetch) and mocks store.write_prices, then checks the provider normalizes -Yahoo's (field, ticker) MultiIndex frame to the store's Open/High/Low/Close/Volume + -DatetimeIndex('Date') shape for both the backadj and unadj adjustments.""" -import sys -import types - -import pandas as pd - - -def _install_fake_yfinance(monkeypatch, download): - mod = types.ModuleType("yfinance") - mod.download = download - monkeypatch.setitem(sys.modules, "yfinance", mod) - - -def _fake_yahoo_frame(): - # yfinance returns a (field, ticker) column MultiIndex even for one symbol. - idx = pd.to_datetime(["2020-01-02", "2020-01-03", "2020-01-06"]) - cols = pd.MultiIndex.from_product( - [["Open", "High", "Low", "Close", "Volume"], ["EEM"]]) - return pd.DataFrame([[10, 11, 9, 10.5, 1000], - [10.5, 11.5, 10, 11, 1200], - [11, 12, 10.8, 11.8, 900]], index=idx, columns=cols) - - -def test_yfinance_update_normalizes_and_writes_both_adjustments(monkeypatch): - from cotdata import store - from cotdata.providers import yfinance as yprov - - _install_fake_yfinance(monkeypatch, lambda *a, **k: _fake_yahoo_frame()) - written = {} - monkeypatch.setattr(store, "write_prices", - lambda sym, adj, df, source: written.__setitem__((sym, adj), (df, source))) - - res = yprov.update(symbols=["MME"]) # MME carries yahoo="EEM" in the registry - assert res["wrote"] == 1 and res["ok"] - assert ("MME", "backadj") in written and ("MME", "unadj") in written # ETF proxy → both - - df, source = written[("MME", "backadj")] - assert source == "yahoo" - assert list(df.columns) == ["Open", "High", "Low", "Close", "Volume"] # flattened - assert df.index.name == "Date" and df.index.tz is None # tz-naive Date index - assert len(df) == 3 and df["Close"].iloc[-1] == 11.8 - - -def test_yfinance_update_skips_symbols_without_ticker(monkeypatch): - from cotdata.providers import yfinance as yprov - # GC has no yahoo ticker → nothing to do, no fetch attempted. - res = yprov.update(symbols=["GC"]) - assert res["wrote"] == 0 and res["ok"] - - -def test_yfinance_update_reports_empty_as_failure(monkeypatch): - from cotdata.providers import yfinance as yprov - _install_fake_yfinance(monkeypatch, lambda *a, **k: pd.DataFrame()) - res = yprov.update(symbols=["MME"]) - assert res["wrote"] == 0 and res["ok"] is False From 117f952f96f55da7f79a4eebacdcd002c5550c1f Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 9 Aug 2026 13:42:20 +0000 Subject: [PATCH 2/2] =?UTF-8?q?Correct=20=C2=A78.5's=20attribution:=20the?= =?UTF-8?q?=20finals=20gate=20landed=20in=20marketdata=20#12?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Written against a stale clone. `--require-final` was wired into `marketdata-update` by PR #12, which merged while this branch was being built, and a duplicate written here converged on the same design independently hours apart. #12 is the better of the two -- it also keeps `--final-cutoff` accepted and ignored, so a scheduler still carrying cotdata's flag does not break on it, and it prints a per-reference defer table. Worth recording rather than quietly fixing: two passes reaching for the same gate is evidence the deletion could not proceed without it, which is the point §8.5 was making. #13 is now only the ported producer tests, which nothing else covers. --- .../2026-08-04-adr7-step2-price-producer-split.md | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/docs/handoffs/2026-08-04-adr7-step2-price-producer-split.md b/docs/handoffs/2026-08-04-adr7-step2-price-producer-split.md index 8735574..444ebaa 100644 --- a/docs/handoffs/2026-08-04-adr7-step2-price-producer-split.md +++ b/docs/handoffs/2026-08-04-adr7-step2-price-producer-split.md @@ -330,16 +330,21 @@ something §7.1 had left: so `cotdata-update --prices --require-final` was the only way to reach it. Deleting that would have left the Windows nightly job ungated, and the failure it prevents is silent — a fetch before Norgate settles writes a provisional bar over a real one with nothing in - the store to say so. Fixed first, as `marketdata` PR #13, because the deletion was not - safe without it. + the store to say so. Fixed in `marketdata` PR #12, which landed + `marketdata-update --bars --domain futures --require-final` (and keeps `--final-cutoff` + accepted-and-ignored so a scheduler carrying cotdata's flag does not break). Worth + recording that #12 and a duplicate written here converged on the same design + independently, hours apart: the gate is not a matter of taste, and a second pass + reaching for it is a signal the deletion could not proceed without it. 2. **Six behaviours had tests only in `cotdata`.** §7.1 ported the provider and not its tests; `marketdata`'s suite covered the pure functions and left everything reachable only through `update()` untested — volume reconstruction, the volume-rank pick, the incremental window, `full=True`, the NDU-down abort, the all-null spec-row skip. Deleting - here would have been the moment those stopped being tested anywhere. Ported in the same - PR. **A file-count check would have missed this entirely**: the test *files* existed on - both sides, with zero name overlap and a real coverage gap underneath. + here would have been the moment those stopped being tested anywhere. Ported as + `marketdata` PR #13. **A file-count check would have missed this entirely**: the test + *files* existed on both sides, with zero name overlap and a real coverage gap + underneath. 3. **Two harnesses read the Norgate store by path.** `scripts/validate_databento_vs_norgate.py` and