diff --git a/CHANGELOG.md b/CHANGELOG.md index 6a1f8b8..a21b7a9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,52 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [Unreleased] +### Removed — BREAKING + +- **The databento provider is gone, and with it every remaining trace of price data.** + ADR-0007 is now fully implemented: `cotdata` is CFTC positioning and nothing else. + databento moved to [`crucible-marketdata`](https://pypi.org/project/crucible-marketdata/) + alongside Norgate and Yahoo, where it writes `bars/futures/databento/` in that + package's store. + + | gone from cotdata | use instead | + |---|---| + | `cotdata-update --ingest-databento` / `--build-databento` / `--reconcile-databento` | the same flags on `marketdata-update` | + | `cotdata-prices` | *(nothing — there is no price half)* | + | `cotdata[databento]` extra | `crucible-marketdata[databento]` | + | `$COTDATA_DATABENTO_RAW` | `$MARKETDATA_DATABENTO_RAW` | + | `store.write_prices` / `read_prices`, `config.prices_dir()` | `marketdata.store.write_bars` / `read_bars` | + | `registry.resolve_source`, `default_price_source`, `PRICE_SOURCES` | marketdata's registry | + | `Symbol.norgate` / `.yahoo` / `.databento` / `.price_source` | marketdata's registry | + | `$COTDATA_PRICE_SOURCE` | `$MARKETDATA_PRICE_SOURCE` | + | `scripts/validate_databento_vs_norgate.py`, `scripts/investigate_databento_roll_rule.py` | same paths in marketdata | + + **This package now has no optional data dependency at all.** CFTC positioning is a + plain HTTP download of public files, so the `databento` extra was the last one and it + left with its provider. + + **The registry lost its vendor columns.** `Symbol` is down to `internal`, + `asset_class`, `is_equity`, `report_type`, `cftc_code` and `hist_codes`. An earlier + note claimed the vendor mappings had to stay because a deployment might share one + registry file between the two packages via `$COTDATA_REGISTRY` — that was wrong, and + is corrected here: this loader hard-requires `cftc_code` and marketdata's equities do + not have one, so the two files can never be the same file. + + **The producer-half machinery is gone**, because there is one producer. `cotdata-cot` + survives as an alias of `cotdata-update` — the scheduled jobs call it by name — but it + no longer scopes anything, and `_HALF_ACTIONS` / `_reject_other_half` are removed. + The manifest is still split (`manifests/cot.json`, `manifests/prices.json`) and the + `prices` and `metadata` domains are still DECLARED: a store built before the move + carries entries there, and an undeclared domain is skipped by `--migrate-manifests`, + which would strand them in the legacy aggregate forever. Read-only history, no writer. + + **What did NOT come across:** databento's dormant per-symbol EOD path + (`fetch_daily_ohlc`, `run_batch_backfill`, `update_all_daily_prices`). It had no + caller anywhere in the fleet, it duplicated the two-stage producer, and the intraday + work it was nominally kept for would need an intraday schema rather than the + `ohlcv-1d` it actually fetched. It remains in this repo's git history. + + ### Removed — BREAKING - **Price bars, and the Norgate and Yahoo producers, are gone from this package** diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 24fbd3b..bb4c0b3 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -79,10 +79,9 @@ There is none here any more. ADR-0007 moved the Norgate integration to [`crucible-marketdata`](https://github.com/mspinola/marketdata), so Norgate changes and their Windows-only constraints belong in that repo's CONTRIBUTING. -Everything in this repo runs cross-platform with no vendor SDK: the CFTC parsers hit -cftc.gov, and the databento provider's tests drive a synthetic raw store rather than the -API. If a test needs a network or a paid key to pass, it does not belong in `tests/` — -put it in `scripts/` and say so in its docstring, as the databento parity harnesses do. +Everything in this repo runs cross-platform with no vendor SDK and no optional extra: +the only source is cftc.gov over plain HTTP. If a test needs a network or a paid key to +pass, it does not belong in `tests/`. ## Code Style diff --git a/README.md b/README.md index 9c823ee..f2837ca 100644 --- a/README.md +++ b/README.md @@ -16,26 +16,25 @@ cotdata separates *fetching* data (a "producer" that talks to vendors) from *usi - **New-data signal.** Every run writes a structured `status.json` so downstream tools can poll one file to detect fresh data. > [!IMPORTANT] -> **Price bars moved out of this package.** Under [ADR-0007](#ecosystem), cotdata is CFTC -> positioning only, and every daily bar — Norgate futures, Yahoo equities/ETFs, the -> `backadj`/`unadj`/`propadj` tiers, contract specifications — now lives in +> **Price bars are not in this package.** Under [ADR-0007](#ecosystem), cotdata is CFTC +> positioning only, and every daily bar — Norgate futures, databento futures, Yahoo +> equities/ETFs, the `backadj`/`unadj`/`propadj` tiers, contract specifications — lives in > [**crucible-marketdata**](https://pypi.org/project/crucible-marketdata/). Replace > `cotdata.get_prices(sym, adjustment)` with `marketdata.get_bars(sym, tier)` and point > `MARKETDATA_STORE` at that store. The two packages keep separate stores and separate > producers; nothing about `get_cot()` changed. -> -> The one price producer still here is **databento**, which has no marketdata equivalent -> yet (see [Cross-platform prices without Norgate](#cross-platform-prices-without-norgate-databento)). ## Data sources at a glance | Data | Source | Cost | Runs on | |------|--------|------|---------| | CFTC COT — legacy / disaggregated / TFF / supplemental | [cftc.gov](https://www.cftc.gov/) | **Free** | any OS | -| Futures prices, back-adjusted | [Databento](https://databento.com/) GLBX.MDP3 | Paid per query | any OS | | *Reading the store* | — | Free | any OS | | Daily bars + contract specs | → [crucible-marketdata](https://pypi.org/project/crucible-marketdata/) | — | — | +No paid dependency, and no optional data extra: this package downloads public CFTC +files over HTTP and reads its own store. + ## Contents - [Quickstart](#quickstart) · [How it works](#how-it-works) · [Reading data](#reading-data-consumer) · [Producing data](#producing-data-producer) · [Windows setup](docs/WINDOWS_SETUP.md) · [Scheduling on Windows](docs/WINDOWS_SCHEDULING.md) · [Scheduling on Linux](docs/LINUX_SCHEDULING.md) · [Syncing the store](docs/SYNCING.md) · [Operations](#operations) · [Concepts & design](#concepts--design) · [COT vintage tracking](#cot-vintage-tracking-as-published-history) · [Reference: schemas](#reference-data-schemas) · [Reference: COT formats](#reference-cot-formats-explained) · [Diagnostics](#diagnostics) · [Development](#development) · [Contributing](#contributing) · [License](#license) @@ -69,14 +68,13 @@ The **store is the API boundary** — not Python imports. Producers write Parque ``` PRODUCER — runs where each source is reachable - CFTC COT download (any OS) databento ingest/build (any OS, paid) - │ │ - └──────────────┬───────────────┘ + CFTC COT download — free, any OS, no vendor SDK + │ ▼ write parquet + manifest ┌────────────────────────────────────────────────────────────┐ │ CANONICAL STORE ($COTDATA_STORE) │ │ cot_legacy/ cot_disagg/ cot_tff/ │ - │ cot_supplemental/ prices/ (databento only) │ + │ cot_supplemental/ │ │ manifests/ status.json │ └────────────────────────────────────────────────────────────┘ │ read (offline, any OS) @@ -96,7 +94,6 @@ The store layout: - `cot_disagg/{symbol}_{code}.parquet` — weekly CFTC Disaggregated positioning. - `cot_tff/{symbol}_{code}.parquet` — weekly CFTC Traders in Financial Futures positioning. - `cot_supplemental/{symbol}_{code}.parquet` — weekly CFTC Supplemental (Commodity Index Trader) positioning, 13 agricultural markets. **Futures-and-options combined**, unlike the three above. -- `prices/{symbol}_{adjustment}.parquet` — **databento-built bars only**, `adjustment` ∈ {`backadj`, `unadj`}. This is the ADR-0006 alternative producer's output, not a general bar store: read Norgate/Yahoo bars from `marketdata` instead. - `manifests/{cot,prices}.json` — per-table `last_date`, `n_rows`, `source`, `updated_at`, `schema_version`, one file per producer half. - `status.json` — machine-readable new-data signal for downstream tools (see [Operations](#operations)). - `vintage/` — optional as-published (vintage) capture: retained raw CFTC downloads plus @@ -149,7 +146,7 @@ longer fills, and stale data is harder to notice than an `AttributeError`. ## Producing data (producer) -Run on the machine that can reach the source. CFTC COT runs anywhere; the optional Databento price build also runs anywhere (see [Cross-platform prices without Norgate](#cross-platform-prices-without-norgate-databento)). Norgate and Yahoo bars are produced by `marketdata-update --bars`, in the [sibling package](https://pypi.org/project/crucible-marketdata/). +Runs anywhere — the CFTC files are a public HTTP download. Bars of every vendor are produced by `marketdata-update` in the [sibling package](https://pypi.org/project/crucible-marketdata/). ```bash COTDATA_STORE=/store cotdata-update --cot-legacy # CFTC Legacy (any OS) @@ -162,57 +159,17 @@ COTDATA_STORE=/store cotdata-vintage fetch # optional: cap Each run prints a per-domain line and a summary footer. A run **exits non-zero** if a fetch hard-fails (CFTC or Databento unreachable), so a scheduler can retry — see [Scheduling on Linux](docs/LINUX_SCHEDULING.md). -### Cross-platform prices without Norgate (databento) - -A server that cannot run Norgate (for example the public dashboard host) can build a price store from Databento instead. One provider owns each symbol end to end, so this is a full replacement, not a blend. Install the extra and set the environment: - -```bash -pip install "cotdata[databento]" - -export COTDATA_STORE=/path/to/store # the store the dashboard reads -export DATABENTO_API_KEY=db-... # for the paid ingest step only -# optional: export COTDATA_DATABENTO_RAW=/path/to/raw # defaults to $COTDATA_STORE/_raw/databento -``` - -Then build the store in order (ingest before build): - -```bash -cotdata-update --ingest-databento # Stage 1 (PAID): raw .n.0/.n.1 ohlcv-1d + statistics -> raw store -cotdata-update --build-databento # Stage 2 (FREE): additive back-adjustment -> $COTDATA_STORE/prices -cotdata-update --cot-all # CFTC COT, the dashboard needs it too -cotdata-update --check # coverage, newest dates, staleness -``` - -- **Two stages, one paid.** Stage 1 is the only step that hits the API. It writes an append-only raw store and resumes from the last fetched date, so re-runs pull only new days. Stage 2 reads that raw store with no API cost, so the back-adjustment can be iterated offline. The raw store is producer-internal, so keep it out of any sync to consumers. -- **History starts 2010-06-06** (the GLBX floor), shallower than Norgate. Markets not on CME Globex (ICE softs, lumber, MSCI intl) are not covered — take those from `marketdata`, which prices them off Yahoo. -- **First-run check.** A healthy symbol prints `built unadj+backadj (N bars, K rolls)`. If it prints `no rolls detected`, back-adjustment is a no-op for that symbol, so investigate before trusting it. -- **Read it back** with `cotdata.store.read_prices(symbol, adjustment)`. There is no consumer bar API here — this store is the alternative producer's output, and the general one is `marketdata.get_bars`. -- **Validate against Norgate** (optional gate) with `scripts/validate_databento_vs_norgate.py`, pointing `--norgate-store` at a `$MARKETDATA_STORE`. -- **Schedule** the two price commands nightly and `--cot-all` weekly — see [Scheduling on Linux](docs/LINUX_SCHEDULING.md). - -> [!NOTE] -> Databento staying here is a deliberate exception to ADR-0007, not an oversight. It has -> no marketdata equivalent yet, and deleting it would destroy a validated alternative -> producer (ADR-0006) plus the only intraday-capable source in the fleet. When it is -> ported, `prices/`, `store.write_prices`/`read_prices` and the `prices` manifest half -> go with it and this package becomes COT-only in full. - -### Producer halves: one host, one job +### One producer, and the manifest split it left behind -`cotdata` has two producers by design: the CFTC downloader and the databento price -builder. Two entry points scope a host to one of them: +`cotdata` used to have two producers — the CFTC downloader and a price producer — and two +entry points that scoped a host to one of them, so a price box could not quietly become a +second COT producer racing the first. Every price producer moved to `marketdata`, so there +is one producer here now and nothing to scope. `cotdata-cot` survives as an alias of +`cotdata-update` because the scheduled jobs call it by name; `cotdata-prices` is gone. -```bash -cotdata-cot --cot-all # CFTC half -cotdata-prices --ingest-databento --build-databento # price half -cotdata-update ... # both, for a single-machine deployment -``` - -Each scoped entry point refuses the other half's flags, so a price box cannot quietly -become a second COT producer racing the first. `--check` and `--reconcile` are read-only -and work from either. - -Each half also owns its own manifest (`manifests/cot.json`, `manifests/prices.json`). +The manifest is still split per half (`manifests/cot.json`, `manifests/prices.json`), and +the `prices` half is now **read-only history**: a store built before the move still carries +those entries, and they have to keep migrating and reconciling rather than being stranded. The legacy top-level `manifest.json` held both halves in ONE file, which was unsafe two ways: the update is a read-modify-write, so two producers lose each other's entries, and a file-level sync between two stores resolves it last-writer-wins and silently discards @@ -238,9 +195,9 @@ The Windows box is also where the **Norgate bar** job runs, but that is now `mar ### Scheduling on Linux (cron) -Full setup, including wrapper scripts, the crontab entries (nightly databento prices, daily COT catch-up, Friday release-window poller), `flock` overlap protection, and troubleshooting (cron's bare environment, timezone conversion, `DATABENTO_API_KEY` not being picked up), is in **[docs/LINUX_SCHEDULING.md](docs/LINUX_SCHEDULING.md)**. +Full setup, including the wrapper script, the crontab entries (daily COT catch-up plus a Friday release-window poller), `flock` overlap protection, and troubleshooting (cron's bare environment, timezone conversion), is in **[docs/LINUX_SCHEDULING.md](docs/LINUX_SCHEDULING.md)**. -The short version: a databento server schedules prices nightly, and COT gets a daily morning catch-up plus a tight Friday-afternoon poll around its ~3:30pm ET release, all idempotent and safe to over-run. +The short version: COT gets a daily morning catch-up plus a tight Friday-afternoon poll around its ~3:30pm ET release, all idempotent and safe to over-run. ### Syncing the store between machines @@ -249,9 +206,11 @@ produced elsewhere. Prefer one producer writing everything and a strictly one-directional sync. (The bar store syncs the same way, separately — it is a different directory with a different producer.) -Two directories must be **excluded** for size: `_cache/` (cotdata's cache of downloaded -CFTC source zips) and `_raw/` (the **paid** databento raw store) are producer-internal -and together are ~70% of the bytes. The legacy `manifest.json` should be excluded too. +One directory must be **excluded** for size: `_cache/`, cotdata's cache of downloaded +CFTC source zips, is producer-internal and free to rebuild. The legacy `manifest.json` +should be excluded too. (A store built before ADR-0007 also has `_raw/` — databento's +paid bronze store — and `prices/`; both belong to `marketdata` now and neither is +written here any more.) Anything a consumer put in the store by hand is a **correctness** issue rather than a saving. No producer creates it, so a mirroring sync deletes it. Exclude it, but the real @@ -315,29 +274,14 @@ COT tables are stored per code as **`{symbol}_{code}`** (e.g. `RTY_23977A`), so ## Concepts & design -### Back-adjusted vs unadjusted prices - -Futures contracts expire, forcing traders to "roll" into the next contract, which usually trades at a slightly different price. Simply stitching contracts together creates artificial price gaps, so two series are stored and a third is derived: - -- **`backadj` (signals & stops).** Gap-free *arithmetic* (additive) rolls shift historical prices to align with the new contract, preserving *absolute* daily point moves. Use this for indicators, signals, and stop-losses to avoid false triggers on rollover gaps. -- **`unadj` (position sizing).** Back-adjustment shifts historical prices (sometimes negative), so you can't use it for dollar values. Use `unadj` (raw, real-life prices) for that day to compute true dollar risk and contract counts. -- **`propadj` (proportional / ratio adjustment).** Derived on read from `unadj` + `backadj`; preserves daily *percentage* returns. Use it for **low-priced, long-history contracts where additive back-adjustment accumulates roll gaps below zero** and breaks price-based stops and R-multiples — see *Class III Milk (DC)* below. - -The databento builder here writes `backadj` and `unadj`. The tier *reader*, including the -derived `propadj`, is `marketdata.get_bars(symbol, tier)` — see that package's README for -the full treatment. Kept here because it is what the databento build produces and how to -judge whether it produced it correctly. - -#### Why `propadj` exists — Class III Milk (DC) - -Norgate publishes continuous futures in only two forms: unadjusted and **additive** back-adjusted (`_CCB`) — there is no native ratio-adjusted series. Additive adjustment subtracts each roll's calendar spread from all prior history, and for a low-priced, seasonal, ~29-year contract like **DC (Class III Milk, ~$15–20/cwt)** those gaps accumulate past zero: **46.7% of `DC_backadj` closes are ≤ 0** (range −9.83 to 23.09). A price-based stop, an R-multiple, or a percentage return is meaningless on a non-positive series, so CMR cannot use DC's `backadj` at all — even though DC is the flagship *new-asset-class* (Dairy) held-out generalization market. - -`propadj` salvages it. Because the additive series `B` and unadjusted series `U` differ by an offset `O = B − U` that steps only at rolls, each roll's calendar spread is recoverable (`s = O[r−1] − O[r]`) and convertible to a multiplicative roll ratio `k = (U[r−1] + s)/U[r−1]`. Scaling each historical segment by the cumulative product of `k` (most-recent segment anchored to actual prices) preserves within-segment percentage returns exactly and is sign-identical to `backadj` on every day including rolls. Note it is **not** strictly positive, as this section once claimed: ratio adjustment scales by a positive factor, so it preserves the underlying's sign — CL's `propadj` close on 2020-04-20 is −24.11 because WTI really settled at −37.63. That is one bar in the whole store, against `backadj`'s 46.7% for DC. Recommendation: **read DC (and any similarly low-priced contract) with `marketdata.get_bars(sym, "propadj")`.** +### Price adjustment tiers -### Providers & authentication - -- **Databento (paid, cross-platform).** A two-stage producer for a server that cannot run Norgate. Stage 1 (`cotdata-update --ingest-databento`) pulls raw `.n.0` / `.n.1` `ohlcv-1d` and `statistics` into an append-only raw store (`$COTDATA_DATABENTO_RAW`, else `_raw/databento` under the store). This is the only paid step, and it is resumable, so re-runs fetch only new dates. Stage 2 (`cotdata-update --build-databento`) derives the back-adjusted prices from the raw store with no API cost, so the build logic can be iterated offline. Set `DATABENTO_API_KEY`. History starts 2010-06-06, and markets not on CME Globex (ICE softs, lumber, MSCI intl) are not covered. The raw store is producer-internal, so exclude it from any consumer sync. -- **Norgate Data / Yahoo Finance** — moved to [crucible-marketdata](https://pypi.org/project/crucible-marketdata/) with the rest of the bar production (ADR-0007), along with the `[norgate]` and `[yahoo]` extras and the `COTDATA_PRICE_SOURCE` resolution that chose between vendors. That per-symbol resolution now lives in marketdata's registry. +Not here. Futures roll, and stitching contracts creates artificial gaps, so bars come in +three tiers — `backadj` for signals and stops, `unadj` for position sizing, `propadj` for +anything measuring percent returns. All three, and the reasoning behind them (including +why `propadj` is not optional: additive back-adjustment drives 46.7% of Class III Milk's +closes below zero), live in +[crucible-marketdata's README](https://github.com/mspinola/marketdata). ### The symbol registry @@ -452,35 +396,18 @@ export COTDATA_STORE=/path/to/synced/store # the shared store uv run pytest # run the tests ``` -For the databento producer, add the extra with `uv pip install -e ".[databento]"`. Use `uv run `, or activate with `source .venv/bin/activate` (Mac/Linux) / `.venv\Scripts\activate` (Windows). +There are no optional data extras — every vendor SDK moved out with the producers. Use `uv run `, or activate with `source .venv/bin/activate` (Mac/Linux) / `.venv\Scripts\activate` (Windows). ## Reference: Data schemas The canonical store uses standard Parquet files. Loaded with `pd.read_parquet()`, they conform to the following schemas. -### Price Data (`prices/{symbol}_{adjustment}.parquet`) - -> [!NOTE] -> **Databento output only.** Norgate/Yahoo bars and the general bar schema (including the -> reconstructed-volume columns and how to read them) moved to -> [crucible-marketdata](https://pypi.org/project/crucible-marketdata/) under ADR-0007. -> What is documented here is what `--build-databento` writes into this store, read back -> with `cotdata.store.read_prices(symbol, adjustment)`. - -Indexed by tz-naive `Date`. The build writes both the back-adjusted (`backadj`) series for signals/stops and the unadjusted (`unadj`) series for true transaction-cost modeling. +### Price Data -**Schema versioning:** `schema_version` in the manifest records the on-disk data version. Consumers key cache invalidation on `cotdata.schema_version()` and can guard with `cotdata.require_schema(min_version)`. - -| Column | Type | Description | -|--------|------|-------------| -| `Date` | DatetimeIndex | Trading day (tz-naive, normalized to midnight). | -| `Open` | float | Opening price. | -| `High` | float | High price. | -| `Low` | float | Low price. | -| `Close` | float | Settlement Close price. | -| `Volume` | float | Continuous contract trading volume (front-month only). | -| `Open Interest` | float | Continuous contract open interest, from the `statistics` schema (stat_type 9). | -| `Delivery Month` | float | Expiration month of the active contract (e.g. `202609`). Used to detect contract rolls. | +Not in this store. Every bar, and the schema describing it, moved to +[crucible-marketdata](https://pypi.org/project/crucible-marketdata/) under ADR-0007. +A store built before that move still has a `prices/` directory: it is history, nothing +here reads or writes it, and `--reconcile` will keep its manifest entries honest. ### Contract Specifications @@ -586,7 +513,7 @@ COT/futures research — crucible is just the most common thing to point at it n Want to contribute or work on cotdata locally? See [CONTRIBUTING.md](CONTRIBUTING.md) for: - Virtual environment setup with `uv` or standard `pip` - Running the test suite -- Platform-specific notes (CFTC parsing and the databento build run anywhere) +- Platform-specific notes (everything here runs anywhere — no vendor SDK) - Code style guidelines ## Contributing diff --git a/docs/LINUX_SCHEDULING.md b/docs/LINUX_SCHEDULING.md index 662e565..e3fee47 100644 --- a/docs/LINUX_SCHEDULING.md +++ b/docs/LINUX_SCHEDULING.md @@ -1,41 +1,30 @@ # Scheduling cotdata on Linux (cron) -For the cross-platform Databento producer path (no Norgate/Windows required) — see [Cross-platform prices without Norgate](../README.md#cross-platform-prices-without-norgate-databento) in the README for the one-time `--ingest-databento` / `--build-databento` setup before automating it here. +> **Only COT is scheduled here now.** ADR-0007 moved every price producer to +> [`marketdata`](https://pypi.org/project/crucible-marketdata/), databento included, so +> the nightly price job on this box is `marketdata-update --ingest-databento` / +> `--build-databento` against `$MARKETDATA_STORE`. See that package's README. The COT +> half below is unchanged. +> +> **Upgrading?** Delete the old `run-prices.sh` — it calls `cotdata-prices`, which no +> longer exists, so the job will fail every night until it is replaced or removed. ## Goal -A databento server schedules **prices nightly** and **COT soon after its Friday ~3:30pm ET release**, with a daily catch-up for holiday delays. Two properties hold: +**COT soon after its Friday ~3:30pm ET release**, with a daily catch-up for holiday delays. Two properties hold: -- **Idempotent.** `--cot-all` HEAD-checks each CFTC year zip and skips it if unchanged. `--ingest-databento` resumes from the last fetched date, so a re-run pulls only new days. Running before new data lands is a harmless no-op. -- **Fails loudly.** A run exits non-zero only on a hard fetch error (source unreachable), not when there is simply nothing new. Because ingest is resumable and COT is idempotent, a failed or missed run is picked up by the next one, so no explicit retry logic is needed. +- **Idempotent.** `--cot-all` HEAD-checks each CFTC year zip and skips it if unchanged. Running before new data lands is a harmless no-op. +- **Fails loudly.** A run exits non-zero only on a hard fetch error (source unreachable), not when there is simply nothing new. Because COT is idempotent, a failed or missed run is picked up by the next one, so no explicit retry logic is needed. ## Wrapper scripts -Cron runs with a bare environment, so put the config and the venv path in a wrapper script (one per command, mirroring the Windows pair). +Cron runs with a bare environment, so put the config and the venv path in a wrapper script. -> **Ready-made templates:** copy [`docs/examples/linux/run-prices.sh`](examples/linux/run-prices.sh) and [`run-cot.sh`](examples/linux/run-cot.sh) out of the repo into your ``, `chmod +x` them, and fill in the placeholders — keep them outside the repo so a `git pull` never clobbers your edited paths. +> **Ready-made template:** copy [`run-cot.sh`](examples/linux/run-cot.sh) out of the repo into your ``, `chmod +x` it, and fill in the placeholders — keep it outside the repo so a `git pull` never clobbers your edited paths. -Inside the scripts, overwrite the plain-text markers: `REPLACE_WITH_STORE_PATH` = your store, `REPLACE_WITH_VENV_PATH` = your virtualenv, `REPLACE_WITH_DATABENTO_KEY` = your Databento key. (They're plain markers, not `<...>` placeholders, because an unedited `<...>` would be read as a shell redirection and the script would fail.) The `` in the crontab lines below is normal fill-in notation. +Inside it, overwrite the plain-text markers: `REPLACE_WITH_STORE_PATH` = your store, `REPLACE_WITH_VENV_PATH` = your virtualenv. (They're plain markers, not `<...>` placeholders, because an unedited `<...>` would be read as a shell redirection and the script would fail.) The `` in the crontab lines below is normal fill-in notation. -`run-prices.sh` — the two-stage databento build: - -```bash -#!/usr/bin/env bash -set -euo pipefail -export COTDATA_STORE=REPLACE_WITH_STORE_PATH -export DATABENTO_API_KEY=REPLACE_WITH_DATABENTO_KEY -BIN=REPLACE_WITH_VENV_PATH/bin/cotdata-prices -"$BIN" --ingest-databento # Stage 1 (paid): raw .n.0/.n.1 to raw store -"$BIN" --build-databento # Stage 2 (free): back-adjusted prices -``` - -Databento is the only price producer left in this package. ADR-0007 moved the Norgate and -Yahoo bar producers to [`marketdata`](https://pypi.org/project/crucible-marketdata/), so the -markets databento does not cover — ICE softs, lumber, the MSCI ETF proxies — are fetched by -`marketdata-update --bars` against `$MARKETDATA_STORE` on whichever box produces that store, -not by this script. - -`run-cot.sh` — COT (note the different command): +`run-cot.sh`: ```bash #!/usr/bin/env bash @@ -44,10 +33,10 @@ export COTDATA_STORE=REPLACE_WITH_STORE_PATH REPLACE_WITH_VENV_PATH/bin/cotdata-cot --cot-all ``` -Make them executable: +Make it executable: ```bash -chmod +x run-prices.sh run-cot.sh +chmod +x run-cot.sh ``` ## Crontab entries @@ -55,10 +44,6 @@ chmod +x run-prices.sh run-cot.sh Add the jobs with `crontab -e`. Cron uses the **server's local** timezone, so convert the ET times below if it is not on Eastern (or set the server to a known zone). `flock` stops a slow run from overlapping the next, and the redirect keeps a log: ```cron -# Prices — nightly (Mon-Sat). GLBX settlements are disseminated the morning after the -# session, so an early-morning run captures the prior session's finalized settlement. -30 6 * * 1-6 flock -n /tmp/cotdata-prices.lock /run-prices.sh >> /prices.log 2>&1 - # COT — daily morning catch-up (holiday-delayed releases and a safety net). 10 8 * * * flock -n /tmp/cotdata-cot.lock /run-cot.sh >> /cot.log 2>&1 @@ -77,19 +62,19 @@ Set `MAILTO=you@example.com` at the top of the crontab to have cron email the ou ### Cron job runs manually but not on schedule -Cron's environment is far barer than an interactive shell — no `PATH` beyond `/usr/bin:/bin`, no `.bashrc`/`.profile` sourced, no venv activation. This is exactly why the wrapper scripts above call the venv's binary by full path (`/bin/cotdata-prices`, `/bin/cotdata-cot`) rather than a bare command name, and `export` every variable instead of relying on a login shell to have set them. If a script works when you run it by hand but not under cron, the first thing to check is whether it depends on something your interactive shell set up implicitly. +Cron's environment is far barer than an interactive shell — no `PATH` beyond `/usr/bin:/bin`, no `.bashrc`/`.profile` sourced, no venv activation. This is exactly why the wrapper script above calls the venv's binary by full path (`/bin/cotdata-cot`) rather than a bare command name, and `export` every variable instead of relying on a login shell to have set them. If a script works when you run it by hand but not under cron, the first thing to check is whether it depends on something your interactive shell set up implicitly. ### Job silently does nothing -Check `/prices.log` or `/cot.log` first — the wrappers redirect both stdout and stderr there. If the log is empty or missing entirely, cron likely never ran the job: check `grep CRON /var/log/syslog` (Debian/Ubuntu) or `journalctl -u cron` (systemd) for the scheduled time to confirm cron invoked it at all. +Check `/cot.log` first — the wrapper redirects both stdout and stderr there. If the log is empty or missing entirely, cron likely never ran the job: check `grep CRON /var/log/syslog` (Debian/Ubuntu) or `journalctl -u cron` (systemd) for the scheduled time to confirm cron invoked it at all. ### Overlapping runs / stale lock -`flock -n` fails fast (doesn't block) if another instance already holds the lock file, so a slow `run-prices.sh` won't stack with the next scheduled run — the second invocation just no-ops and exits. The lock releases automatically when the holding process exits, including on a crash, so a stale lock that blocks forever generally indicates a *hung*, still-running process, not manual cleanup — check with `ps aux | grep cotdata` before deleting anything under `/tmp`. +`flock -n` fails fast (doesn't block) if another instance already holds the lock file, so a slow `run-cot.sh` won't stack with the next scheduled run — the second invocation just no-ops and exits. The lock releases automatically when the holding process exits, including on a crash, so a stale lock that blocks forever generally indicates a *hung*, still-running process, not manual cleanup — check with `ps aux | grep cotdata` before deleting anything under `/tmp`. ### Permission denied running the wrapper -Confirm `chmod +x` was applied to both `.sh` files, and that the shebang (`#!/usr/bin/env bash`) resolves — run `which bash` to confirm it's on `PATH` for the cron user (usually is, but matters more on minimal containers). +Confirm `chmod +x` was applied to the `.sh` file, and that the shebang (`#!/usr/bin/env bash`) resolves — run `which bash` to confirm it's on `PATH` for the cron user (usually is, but matters more on minimal containers). ### Timezone confusion on the Friday window diff --git a/docs/SYNCING.md b/docs/SYNCING.md index cfb59f7..c456e98 100644 --- a/docs/SYNCING.md +++ b/docs/SYNCING.md @@ -7,14 +7,18 @@ This page is about **what** to move and what to leave behind. The transport is t part and comes last. > [!NOTE] -> **Two stores since ADR-0007.** Daily bars moved to +> **Two stores since ADR-0007.** Every bar — Norgate, databento and Yahoo alike — moved to > [`crucible-marketdata`](https://pypi.org/project/crucible-marketdata/) and its own > `$MARKETDATA_STORE`. The Windows box still produces both and still pushes both to the same > replicas, so the topology, exclusions, auth gotchas and preflight advice below apply > unchanged — there are now **two source directories to mirror instead of one**, and the -> `prices/`, `metadata/` rows in the exclusion table describe the bar store rather than this -> one. Where a command names `cotdata-prices --prices`, read +> `bars/`, `metadata/` and `_raw/` rows describe the bar store rather than this one. Where a +> command names `cotdata-prices --prices`, read > `marketdata-update --bars --domain futures --require-final`. +> +> A cotdata store built before the move still has `prices/`, `metadata/` and `_raw/` sitting +> in it. Nothing writes them any more; leaving them is harmless and deleting them is safe +> once the bar store is confirmed synced. ## This deployment: one Norgate producer, two replicas A single Windows server is the only producer (Norgate prices, CFTC COT). It feeds two @@ -72,12 +76,15 @@ maintenance than a per-symbol roll-rule table. give it an explicit writable `-o UserKnownHostsFile=`, and use cygdrive (`/cygdrive/c/…`) paths throughout, including the key. The example script wires all three. -**Provider cutover (one-time).** The server previously held a databento-built store, so its -`prices/` and `manifests/prices.json` carry databento data under the very keys the Norgate -push writes. `sync_preflight.py` will (correctly) refuse: it sees the same -`prices/_.parquet` produced by a different source on each side, the 94-collision -case from "Check before you mirror". That refusal is the tool working, not a -misconfiguration. Resolve it once by clearing the server's `prices/` and `manifests/` +**Provider cutover (one-time, and now historical).** The server previously held a +databento-built store, so its `prices/` and `manifests/prices.json` carry databento data +under the very keys the Norgate push writes. `sync_preflight.py` will (correctly) refuse: it +sees the same `prices/_.parquet` produced by a different source on each side, the +94-collision case from "Check before you mirror". That refusal is the tool working, not a +misconfiguration. **ADR-0007 removed the possibility**: the vendor is a directory in +marketdata's layout (`bars/futures/norgate/` beside `bars/futures/databento/`), so two +vendors cannot contend for one path at all. Resolve the legacy case once by clearing the +server's `prices/` and `manifests/` before the first Norgate push, so the store is rebuilt as a clean Norgate replica; every push after that is a same-source mirror with nothing to collide. @@ -114,20 +121,19 @@ about two producers writing the *same* files. This is the part that matters, and on a real store it is most of the bytes. -Per store, since there are now two. `bars/` and `metadata/` live in `$MARKETDATA_STORE`; -everything else here is `$COTDATA_STORE`. `prices/` appears in the cotdata store only when -the databento producer is in use. +Per store, since there are now two. `bars/`, `metadata/` and `_raw/` live in +`$MARKETDATA_STORE`; everything else here is `$COTDATA_STORE`. | Directory | Sync? | Why | |---|---|---| | `bars/` (marketdata) | **yes** | the data | | `metadata/` (marketdata) | **yes** | contract specs | | `cot_legacy/`, `cot_disagg/`, `cot_tff/` | **yes** | the data | -| `prices/` | **yes**, if you run databento | the ADR-0006 alternative producer's output | +| `prices/` | **NO** (legacy) | pre-ADR-0007 leftover; bars live in the marketdata store now | | `manifests/` | **yes** | per-half bookkeeping | | `status.json` | yes | the producer's own view, useful on the replica | | `_cache/` | **NO** | cotdata's own download cache of CFTC source zips, producer-internal, free to rebuild | -| `_raw/` | **NO** | databento append-only raw store, producer-internal | +| `_raw/` | **NO** | databento's append-only PAID raw store, producer-internal (marketdata's now) | | anything a consumer added by hand | **NO** | no producer creates it, so a mirror deletes it (see below) | | `manifest.json` | **NO** | legacy aggregate, nothing writes it (see below) | diff --git a/docs/WINDOWS_SCHEDULING.md b/docs/WINDOWS_SCHEDULING.md index 1342f1a..3a0fef2 100644 --- a/docs/WINDOWS_SCHEDULING.md +++ b/docs/WINDOWS_SCHEDULING.md @@ -3,7 +3,7 @@ New to Python and cotdata on Windows? Start with the [Windows Setup Guide](WINDOWS_SETUP.md) — install Python, create the venv, and confirm `cotdata-update --cot-legacy` works by hand before automating it. > [!IMPORTANT] -> **The price task now runs a different package.** ADR-0007 moved Norgate bar production to +> **The price task runs a different package.** ADR-0007 moved ALL bar production to > [`crucible-marketdata`](https://pypi.org/project/crucible-marketdata/), so the nightly job is > `marketdata-update --bars --domain futures --require-final` against `$MARKETDATA_STORE`, not > `cotdata-prices --prices`. It is still scheduled here, on the same box, at the same time, @@ -12,8 +12,8 @@ New to Python and cotdata on Windows? Start with the [Windows Setup Guide](WINDO > store variable moved. > > **Upgrading?** Edit `run-prices.cmd` to the new command and give it `MARKETDATA_STORE`. A -> wrapper still calling `cotdata-prices --prices` now dies on an unrecognised flag, which -> Task Scheduler shows as a failed run — loud, not silent. +> wrapper still calling `cotdata-prices` now fails to resolve the command at all — that entry +> point no longer exists — which Task Scheduler shows as a failed run. Loud, not silent. ## Goal @@ -24,7 +24,7 @@ New to Python and cotdata on Windows? Start with the [Windows Setup Guide](WINDO ## Wrapper scripts -Create **two** wrapper scripts — they run *different* commands from *different* packages: `marketdata-update` for the bars, `cotdata-cot` for the COT half. cotdata's own entry points stay half-scoped (`cotdata-cot` / `cotdata-prices`) and each refuses the other half's flags, so a host is scoped to one job and a price box cannot quietly become a second COT producer. +Create **two** wrapper scripts — they run *different* commands from *different* packages: `marketdata-update` for the bars, `cotdata-cot` for the COT. (`cotdata-cot` is an alias of `cotdata-update`; it used to be half of a scoped pair, and the other half went with the price producers.) > **Ready-made templates:** copy [`docs/examples/windows/run-prices.cmd`](examples/windows/run-prices.cmd) and [`run-cot.cmd`](examples/windows/run-cot.cmd) out of the repo into your `` (e.g. `C:\Users\you\cotdata\scheduler\`) rather than retyping them — then just fill in the placeholders. Keep them outside the repo so a `git pull` never clobbers your edited paths. diff --git a/docs/WINDOWS_SETUP.md b/docs/WINDOWS_SETUP.md index f4d02e5..c7da504 100644 --- a/docs/WINDOWS_SETUP.md +++ b/docs/WINDOWS_SETUP.md @@ -142,10 +142,10 @@ Should print a version number. Confirm the CLI resolves too: ```cmd cotdata-update --help ``` -If `--cot-all` and `--build-databento` appear, you are set. If `import` errors instead, your -venv isn't activated (look for `(.venv)`/`(cotdata)` in the prompt). If you see `--prices` or -`--metadata`, you are on a pre-0.4.0 build from before the ADR-0007 split — those flags are -gone, and Norgate bars come from `marketdata-update --bars`. +If `--cot-all` appears, you are set. If `import` errors instead, your venv isn't activated +(look for `(.venv)`/`(cotdata)` in the prompt). If you see `--prices`, `--metadata` or any +`--*-databento` flag, you are on a pre-0.5.0 build from before the ADR-0007 split — every +price producer moved to `marketdata`. ## Step 4: Set Environment Variables @@ -321,7 +321,7 @@ If you'd rather keep the `pip`-installed copy, add its `Scripts` directory to `P ### The install succeeded but a console script is missing Symptom: `pip install -e .` printed success and exited 0, but -`.venv\Scripts\cotdata-prices.exe` (or `cotdata-cot.exe`, or `cotdata-update.exe` +`.venv\Scripts\cotdata-cot.exe` (or `cotdata-update.exe` after a fresh setup) is not there. **Cause: a `uv venv` has no `pip` of its own.** So a bare `pip install` inside an diff --git a/docs/databento_norgate_parity.md b/docs/databento_norgate_parity.md deleted file mode 100644 index b67fe25..0000000 --- a/docs/databento_norgate_parity.md +++ /dev/null @@ -1,112 +0,0 @@ -# databento vs Norgate parity (ADR-0006) - -Status of the databento back-adjusted price build measured against the Norgate build, -and what still blocks promoting databento to a full drop-in provider. - -## Why this exists - -ADR-0006 builds each provider's continuous series independently (single provider per -symbol, no cross-vendor stitching). The absolute price LEVELS are allowed to differ -because each provider's roll calendar and back-adjust anchor float on their own. What -must agree is the SHAPE: additive (Panama) back-adjustment preserves absolute daily -price changes, so `Close.diff()` on any non-roll day should match between the two -builds. `scripts/validate_databento_vs_norgate.py` quantifies that per symbol -(change correlation, scale ratio, roll-date agreement). - -## The run - -Built all 40 databento-capable symbols into a separate server store and compared -`backadj` against the local Norgate store over the databento history (2010-06 to -present): - -``` -COTDATA_STORE=~/code/cotdata_store_server \ -COTDATA_DATABENTO_RAW=~/code/cotdata_store/_raw/databento \ -cotdata-update --build-databento - -python scripts/validate_databento_vs_norgate.py \ - --norgate-store ~/code/marketdata_store \ - --databento-store ~/code/cotdata_store_server \ - --symbols <...> -``` - -> **Since ADR-0007 step 2** the Norgate side of this comparison is a `$MARKETDATA_STORE` -> (`bars/futures/norgate/`), not a cotdata store — the run above was made before the split -> and its `--norgate-store` pointed at `~/code/cotdata_store`. Both harnesses read either -> layout, so the numbers below reproduce against either copy. The databento side is -> unchanged: it still writes `prices/` under `$COTDATA_STORE`. - -## Result: three buckets - -| Bucket | N | Symbols | Reading | -|---|---|---|---| -| Clean | 14 | ES NQ YM RTY EMD NKD GC PL PA 6E 6B 6C BTC ETH | change corr >= 0.996, scale ~1.0. Ready. | -| Units (x100) | 3 | SI HG 6J | Shape agrees (corr >0.998). Pure unit difference. Fixed. | -| Roll-cal minor | 12 | HO ZL ZW KE 6A 6S 6M 6N ZN ZT ZF ZB | corr 0.95 to 0.994. Borderline. | -| Roll-cal MAJOR | 11 | CL RB NG ZC ZS ZM ZO DC LE HE GF | corr 0.33 to 0.94. The blocker. | - -The clean bucket is the quarterly-roll financials, metals, and crypto. The two -problem buckets are the monthly-contract commodities and livestock. - -## Finding 1: units (SI, HG, 6J), resolved - -databento reports true dollars (silver $58/oz, copper $4.5/lb, JPY $0.0061). The -toolchain's convention, inherited from Norgate, quotes silver and copper in cents and -JPY in the IMM x100 form, so those three came out 100x off in both unadj and backadj. -The daily-change correlation already sat above 0.998, confirming it was units only, -not the back-adjustment. - -Fixed in the build with a small deny-by-default `_PRICE_SCALE` allowlist -(`{"SI": 100, "HG": 100, "6J": 100}`) applied to the price columns before the roll-gap -math, never to Volume or Open Interest. After a rebuild, SI/HG/6J scale ratio is ~1.0 -and SI passes parity outright. HG and 6J still sit just under the strict 0.999 -correlation gate, but that residual is the roll-calendar difference below, not units. - -## Finding 2: roll calendar (the monthly-roll set), open - -databento's `.n.0` continuous rolls on its own open-interest rule (read from the -`instrument_id` change). For the monthly-contract commodities and livestock that places -rolls on almost entirely different dates than Norgate (near-zero common roll dates), so -between the two roll points the two series track different delivery months whose daily -changes genuinely differ. The back-adjusted shape then diverges: CL 0.78, HE 0.67, -DC 0.33. This is not a bug, it is a different continuous methodology, but it means -databento CL is a materially different series from Norgate CL, so a book validated on -one would behave differently on the other. - -### Hypothesis - -databento offers three continuous roll rules, chosen by the middle letter of the -continuous symbol: `c` (calendar, roll on expiration), `n` (open interest, current -choice), `v` (volume). Norgate's monthly-commodity rolls may line up far better with -`v` or `c` than with `n`. - -### How to test it - -`scripts/investigate_databento_roll_rule.py` pulls each rule's daily bars (ohlcv only, -so it is a cheap one-request-per-rule pull, no statistics), reads the roll dates off the -`instrument_id` changes, and scores each rule against Norgate's roll dates (from the -`Delivery Month` column). Needs `DATABENTO_API_KEY` and a Norgate-built store: - -``` -DATABENTO_API_KEY=... python scripts/investigate_databento_roll_rule.py \ - --norgate-store ~/code/marketdata_store \ - --symbols CL ZS NG HE ZC LE --tol-days 3 -``` - -If one rule beats `.n` broadly, switch the producer's `_FEEDS` root (and re-ingest the -affected symbols). If different symbols favor different rules, add a per-symbol -roll-rule field to the registry rather than a single global change. - -## Decision (2026-07-27) - -The roll-rule investigation is complete. It is per-symbol, not global: energy (CL, NG) tracks -Norgate on the calendar roll `.c` (near-perfect, 0.995 to 1.000 roll-date match), grains (ZS, -ZC) on the volume roll `.v` (best but loose, a few days off each roll, so about 0.95 not 0.999), -and the quarterly financials and metals already match on the open-interest roll `.n`. - -That per-symbol roll rule is documented here but intentionally NOT built. The server will source -Norgate via a Windows-to-Linux store sync rather than databento. That is risk-free (the dashboard -then shows exactly what local research shows, the same Norgate data) and lower maintenance than -carrying a per-symbol roll-rule table plus a grain series that never quite matches. databento -remains a validated, provider-different alternative source, not the server's price provider. -ADR-0006 is Accepted on that basis (see its Outcome section). diff --git a/docs/design/finals_ready_data_driven.md b/docs/design/finals_ready_data_driven.md index 0ff979e..7495d21 100644 --- a/docs/design/finals_ready_data_driven.md +++ b/docs/design/finals_ready_data_driven.md @@ -10,7 +10,8 @@ the right moment against the live feed. `marketdata-update --bars --domain futures --require-final`), because ADR-0007 step 2 moved bar production out of `cotdata` — §7.1 ported the code and §7.5 deleted the copy here. The design is unchanged and the reasoning below still holds; only the package and the CLI moved. -Two details this document states are now out of date on their face: `--final-cutoff` no +Three details this document states are now out of date on their face: `cotdata-prices` +no longer exists (the entry point went with the last price producer), `--final-cutoff` no longer exists in either package (§107 kept it as an accepted-but-ignored fallback; §7.5 removed it with the rest of the flags), and the file path in the Problem section is `cotdata`'s, which is gone. Left as written, per the doc lifecycle: this is a point-in-time diff --git a/docs/design/reading-the-store.md b/docs/design/reading-the-store.md index a8b2af1..ba4a4b2 100644 --- a/docs/design/reading-the-store.md +++ b/docs/design/reading-the-store.md @@ -124,14 +124,16 @@ Three facts that were each written down separately and never composed: broadened toward parity with Norgate". **Composed: a databento-backed futures store cannot produce `propadj` at all**, so any -consumer needing correct percentage returns cannot be served by one. This is a live constraint -rather than a historical note, and it is a **tier fact, not an operating system fact**, a -distinction that has been got wrong more than once: - -Step 2 §7.5 sharpened it into a structural one. `marketdata` now refuses to derive `propadj` -unless BOTH stored tiers are present, rather than returning an empty frame — the store cannot -hand back a silently wrong series. databento, which still writes into `$COTDATA_STORE`, does -write both tiers, but there is no bar reader here that derives the third from them. +consumer needing correct percentage returns cannot be served by one. It is a **tier fact, not +an operating system fact**, a distinction that has been got wrong more than once: + +**Fact 3 is now out of date, and the constraint with it.** ADR-0007 scoped databento to one +`backadj` series per symbol for the Linux dashboard. When it was ported into `marketdata` +(2026-08-09) it came across as a full futures provider writing BOTH stored tiers, because +that package enforces both-tiers-or-neither on every producer — `propadj` derives from the +pair, and `get_bars` raises rather than returning empty when only one is present. So a +databento-backed store now produces `propadj` like any other. The §5 table below is kept as +written because the OS/tier distinction it draws is still the point. | box | Python >= 3.10 | Norgate | can produce a store carrying all tiers | |---|---|---|---| diff --git a/docs/examples/linux/run-prices.sh b/docs/examples/linux/run-prices.sh deleted file mode 100755 index f5ce770..0000000 --- a/docs/examples/linux/run-prices.sh +++ /dev/null @@ -1,20 +0,0 @@ -#!/usr/bin/env bash -# cotdata price update wrapper for cron / systemd (Linux, databento producer). -# Copy this file next to your crontab dir and overwrite the markers below: -# REPLACE_WITH_STORE_PATH = your data store e.g. /srv/cotdata_store -# REPLACE_WITH_DATABENTO_KEY = your Databento API key e.g. db-... -# REPLACE_WITH_VENV_PATH = your cotdata venv e.g. /opt/cotdata/.venv -# (Plain-text markers, not angle-bracket placeholders: an unedited <...> would be -# read as a shell redirection. Stage 1 is the paid pull, Stage 2 a free rebuild.) -# cotdata-prices runs the PRICE half only and refuses --cot-all. -# Databento is the only price producer left here: ADR-0007 moved the Norgate and -# Yahoo bar producers to `marketdata`, so the markets databento does not cover -# (ICE softs, lumber, MSCI proxies) come from `marketdata-update --bars` against -# $MARKETDATA_STORE, not from this script. -# See docs/LINUX_SCHEDULING.md for the crontab and flock setup. -set -euo pipefail -export COTDATA_STORE=REPLACE_WITH_STORE_PATH -export DATABENTO_API_KEY=REPLACE_WITH_DATABENTO_KEY -BIN=REPLACE_WITH_VENV_PATH/bin/cotdata-prices -"$BIN" --ingest-databento # Stage 1 (paid): raw .n.0/.n.1 to raw store -"$BIN" --build-databento # Stage 2 (free): back-adjusted prices diff --git a/docs/examples/mac/pull-store.sh b/docs/examples/mac/pull-store.sh index a32a71f..6aa1da1 100755 --- a/docs/examples/mac/pull-store.sh +++ b/docs/examples/mac/pull-store.sh @@ -20,7 +20,8 @@ DEST="REPLACE_WITH_LOCAL_STORE" # What each exclusion is for: # _cache, _raw producer-internal, most of the bytes. _cache is cotdata's cache -# of downloaded CFTC zips; _raw is the PAID databento raw store. +# of downloaded CFTC zips. _raw is a pre-ADR-0007 leftover (databento's +# paid bronze store, now marketdata's) — still excluded. # citpy consumer-owned, not written by any producer, so --delete removes # it and no producer run brings it back. Kept as a backstop: such # files belong outside the store. See docs/SYNCING.md. diff --git a/docs/examples/windows/push-to-server.cmd b/docs/examples/windows/push-to-server.cmd index 253e250..c49a6ba 100644 --- a/docs/examples/windows/push-to-server.cmd +++ b/docs/examples/windows/push-to-server.cmd @@ -50,7 +50,8 @@ set "SSH=%SSH_EXE% -i %KEY% -o BatchMode=yes -o StrictHostKeyChecking=accept-new REM Data first, manifests last, so a manifest never announces parquet that has not REM landed (harmless if reversed; readers open parquet directly). --delete makes REM this a true mirror. The exclusions match the Mac push: -REM _cache, _raw producer-internal; _raw/databento (the paid databento bronze) +REM _cache, _raw producer-internal. _raw/databento (the paid databento bronze) is a +REM pre-ADR-0007 leftover here, owned by marketdata now REM rides under _raw and so is excluded, per ADR-0006. REM citpy consumer-owned on the server; excluding it from --delete is REM what stops the mirror from wiping it. diff --git a/docs/examples/windows/sync-store.cmd b/docs/examples/windows/sync-store.cmd index 97cea07..4c3be7b 100644 --- a/docs/examples/windows/sync-store.cmd +++ b/docs/examples/windows/sync-store.cmd @@ -15,7 +15,8 @@ setlocal REM /MIR mirrors (copies new + deletes removed). /XD excludes directories: REM _cache, _raw producer-internal, ~70%% of the bytes. _cache is cotdata's -REM cache of downloaded CFTC zips; _raw is the PAID databento store. +REM cache of downloaded CFTC zips. _raw is a pre-ADR-0007 leftover +REM (databento's paid bronze store, now marketdata's) — still excluded. REM citpy consumer-owned, not written by any producer, so /MIR removes it REM and no producer run brings it back. Kept as a backstop: such REM files belong outside the store. See docs/SYNCING.md. diff --git a/docs/handoffs/2026-08-04-adr7-step2-price-producer-split.md b/docs/handoffs/2026-08-04-adr7-step2-price-producer-split.md index 444ebaa..9dc377e 100644 --- a/docs/handoffs/2026-08-04-adr7-step2-price-producer-split.md +++ b/docs/handoffs/2026-08-04-adr7-step2-price-producer-split.md @@ -362,3 +362,40 @@ purpose; §8.2 already voided the `crowdmon` repoint on the same grounds. now has a second decision to record — that databento remains in `cotdata` with no marketdata equivalent, which is a live exception to the ADR's own boundary rather than a step still to do. + +### 8.6 §7.5 finished: databento ported, 2026-08-09 + +§8.5 recorded databento as the one thing §7.5 left, and framed it as "outstanding work, +not a settled boundary". It is done. `crucible-stack` ADR-0007 is fully implemented and +`cotdata` is CFTC positioning with no price surface of any kind. + +**What moved:** the live two-stage producer (paid resumable ingest, the batch variant, +the both-directions reconcile of the resume ledger, the free offline build), both parity +harnesses and their tests. + +**What did not:** the dormant per-symbol EOD path (`fetch_daily_ohlc`, +`run_batch_backfill`, `update_all_daily_prices`). Its docstring kept it for "the intraday +news-failure work" and the code does not support that claim — it fetches `ohlcv-1d`, so +it is a DAILY path duplicating the two-stage producer, with a parallel cache and a +yfinance fallback. A fleet sweep found no caller; npf's same-named function is its own +shim over `get_bars`. Left in git history. + +**One rule had to change rather than move.** cotdata defaults a symbol's `databento` root +to its internal symbol unconditionally, which is safe with ONE domain. marketdata has +two, and that default would hand every equity a GLBX root and let `resolve_source` route +SPY to a vendor that cannot serve it. So the default is futures-only there. This is the +third time this port has turned up a rule that was correct in a single-domain package and +wrong in a two-domain one; the first two were the store's tier axis (one frame per symbol +cannot hold `backadj` and `unadj`) and the manifest's vendor key. + +**A claim in §7.5 was wrong and is corrected in the code.** A comment left in +`registry.py` justified keeping the vendor columns on the grounds that a deployment might +SHARE one registry file between the two packages via `$COTDATA_REGISTRY`. It cannot: +cotdata's loader hard-requires `cftc_code` and marketdata's equities have none. The +columns are gone. + +**What this removes from the fleet's surface:** cotdata now has no optional data +dependency, no vendor SDK, no API key, and no producer-half machinery — `cotdata-prices` +is gone and `cotdata-cot` is a bare alias kept because the scheduled jobs call it by +name. `prices`, `metadata` and `_raw/` survive in real stores as history that +`--migrate-manifests` and `--reconcile` still handle, and nothing writes. diff --git a/pyproject.toml b/pyproject.toml index 055b1fe..032f831 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "cotdata" -version = "0.4.0" +version = "0.5.0" description = "Canonical CFTC positioning (COT) data layer — a producer/consumer split over a file-based store." readme = "README.md" authors = [ @@ -31,18 +31,17 @@ Repository = "https://github.com/mspinola/cotdata" Issues = "https://github.com/mspinola/cotdata/issues" [project.optional-dependencies] -# Cross-platform price producer: two-stage ingest/build back-adjustment (see the -# Providers section of the README and ADR-0006). Also serves the intraday work. -# The `norgate` and `yahoo` extras moved to crucible-marketdata with the bar -# producers they installed for (ADR-0007). -databento = ["databento>=0.30"] - +# No vendor extras. `norgate`, `yahoo` and `databento` all moved to +# crucible-marketdata with the producers they installed for (ADR-0007), so this +# package has no optional data dependency at all: CFTC positioning is a plain HTTP +# download of public files. dev = ["pytest>=7", "ruff==0.15.22"] [project.scripts] cotdata-update = "cotdata.update:main" +# An alias for cotdata-update, kept because scheduled jobs call it by name. There is +# no `cotdata-prices` any more: it ran the price half, and there is no price half. cotdata-cot = "cotdata.update:main_cot" -cotdata-prices = "cotdata.update:main_prices" cotdata-vintage = "cotdata.vintage_cli:main" cotdata-schedule = "cotdata.vintage_cli:main_schedule" diff --git a/requirements.txt b/requirements.txt index 2cf1db1..17c51e5 100644 --- a/requirements.txt +++ b/requirements.txt @@ -7,8 +7,7 @@ requests>=2.28 python-dateutil>=2.8 xlrd>=2.0 -# Bars are NOT here: the Norgate and Yahoo producers moved to crucible-marketdata -# (ADR-0007), and so did their `norgate` / `yahoo` extras. -# The databento price provider stays (ADR-0006 alternative producer + the intraday -# work) and needs its optional extra: -# uv pip install -e ".[databento]" +# Bars are NOT here, and neither is any vendor SDK. Every price producer -- Norgate, +# Yahoo and databento -- moved to crucible-marketdata (ADR-0007), taking the `norgate`, +# `yahoo` and `databento` extras with them. This package has no optional data +# dependency: CFTC positioning is a plain HTTP download of public files. diff --git a/scripts/investigate_databento_roll_rule.py b/scripts/investigate_databento_roll_rule.py deleted file mode 100644 index 30ed2bb..0000000 --- a/scripts/investigate_databento_roll_rule.py +++ /dev/null @@ -1,196 +0,0 @@ -#!/usr/bin/env python -"""ADR-0006 follow-up: which databento continuous roll rule tracks Norgate? - -The parity check (validate_databento_vs_norgate.py) found that databento's `.n.0` -continuous (open-interest roll) diverges from Norgate for the monthly-contract -commodities and livestock: the two place their rolls on almost entirely different -dates, so the back-adjusted daily-change shape disagrees (CL corr 0.78, HE 0.67, -DC 0.33). The financial and metals symbols that roll quarterly agree fine. - -databento offers three continuous roll rules, selected by the middle letter of the -continuous symbol: - c = calendar (roll on the expiration calendar) - n = open interest (what the producer currently uses) - v = volume (roll when the next contract's volume takes over) - -This spike pulls each rule's daily bars for a few symbols (ohlcv only, so it is -cheap: one small request per rule, no statistics), reads the roll dates off the -`instrument_id` changes, and scores each rule against Norgate's own roll dates -(from the `Delivery Month` column in the Norgate store). The rule with the highest -match rate and smallest date offset is the candidate to switch `_FEEDS` to. - -It reads the paid API, so it is NOT run in CI and is not wired into the producer. -Run it on a machine with DATABENTO_API_KEY set and a Norgate-built store to read. - -The slow part is the API pull, so roll dates are cached per (root, rule) on disk: a -re-run, or extra --tol-days values, cost no extra pulls. Pull a symbol once, then sweep -tolerances for free. - -Since ADR-0007 the Norgate store is a $MARKETDATA_STORE (`bars/futures/norgate/`); -the pre-move cotdata `prices/` layout is still read, so an older synced copy works. - -Usage: - DATABENTO_API_KEY=... python scripts/investigate_databento_roll_rule.py \ - --norgate-store ~/code/marketdata_store \ - --symbols CL ZS NG HE --tol-days 3 7 10 # sweep, one pull per rule -""" -from __future__ import annotations - -import argparse -import json -import sys -from pathlib import Path - -import pandas as pd - -from cotdata.providers.databento import GLBX_HISTORY_FLOOR, _client_from_env, _fetch -from cotdata.registry import all_symbols - -_DEFAULT_SYMBOLS = ["CL", "ZS", "NG", "HE"] -_DEFAULT_RULES = ["c", "n", "v"] - - -def norgate_roll_dates(store: str, symbol: str) -> pd.DatetimeIndex: - """Roll dates from a Norgate-built store: the first session on each new front - contract, read from the `Delivery Month` column of the backadj parquet.""" - for layout in ("bars/futures/norgate", "prices"): # marketdata, then pre-ADR-0007 - p = Path(store) / layout / f"{symbol}_backadj.parquet" - if p.exists(): - break - else: - return pd.DatetimeIndex([]) - df = pd.read_parquet(p) - if "Delivery Month" not in df.columns: - return pd.DatetimeIndex([]) - idx = pd.to_datetime(df.index).tz_localize(None).normalize() - dm = df["Delivery Month"].astype(str) - is_new = dm.ne(dm.shift(1)) & dm.shift(1).notna() - return pd.DatetimeIndex(idx[is_new.values]) - - -def databento_roll_dates(client, dataset: str, root: str, rule: str) -> pd.DatetimeIndex: - """Roll dates for one continuous roll rule: the first session on each new - `instrument_id` in the `{root}.{rule}.0` daily bars.""" - end = (pd.Timestamp.now().normalize() - pd.Timedelta(days=1)).strftime("%Y-%m-%d") - raw = _fetch(client, dataset, f"{root}.{rule}.0", "ohlcv-1d", GLBX_HISTORY_FLOOR, end) - if raw is None or raw.empty or "instrument_id" not in raw.columns: - return pd.DatetimeIndex([]) - idx = pd.to_datetime(raw.index).tz_convert(None).normalize() - df = pd.DataFrame({"iid": raw["instrument_id"].values}, index=idx).sort_index() - df = df[~df.index.duplicated(keep="last")] - is_new = df["iid"].ne(df["iid"].shift(1)) & df["iid"].shift(1).notna() - return pd.DatetimeIndex(df.index[is_new.values]) - - -def roll_dates_cached(client, dataset, root, rule, cache_dir: Path, refresh: bool): - """databento_roll_dates with an on-disk cache. The API pull is the only slow part and - roll dates are historical (a past roll never moves), so once pulled a (root, rule) is - reused across tolerance sweeps and re-runs. `--refresh` forces a fresh pull.""" - cache_dir.mkdir(parents=True, exist_ok=True) - fp = cache_dir / f"{root}.{rule}.roll.json" - if fp.exists() and not refresh: - rolls = json.loads(fp.read_text()).get("rolls", []) - return pd.DatetimeIndex(pd.to_datetime(rolls)), True # (dates, from_cache) - dates = databento_roll_dates(client, dataset, root, rule) - fp.write_text(json.dumps({"root": root, "rule": rule, "dataset": dataset, - "rolls": [str(d.date()) for d in dates]})) - return dates, False - - -def score(dbrolls: pd.DatetimeIndex, ngrolls: pd.DatetimeIndex, tol_days: int): - """Of the Norgate rolls that fall inside databento's covered span, how many have a - databento roll within tol_days? Norgate rolls before databento's history floor have no - possible match and are excluded; databento rolls are NOT clipped, so a Norgate roll near - the edge can still match a databento roll just outside the window. - Returns (n_db, n_ng_in_span, matched, match_rate, median_abs_offset_days).""" - if len(dbrolls) == 0 or len(ngrolls) == 0: - return len(dbrolls), 0, 0, float("nan"), float("nan") - db_sorted = dbrolls.sort_values() - ng = ngrolls[(ngrolls >= db_sorted.min()) & (ngrolls <= db_sorted.max())] - if len(ng) == 0: - return len(db_sorted), 0, 0, float("nan"), float("nan") - offsets, matched = [], 0 - for d in ng: - nearest = min((abs((d - x).days) for x in db_sorted), default=None) - if nearest is not None and nearest <= tol_days: - matched += 1 - offsets.append(nearest) - rate = matched / len(ng) - med = float(pd.Series(offsets).median()) if offsets else float("nan") - return len(db_sorted), len(ng), matched, rate, med - - -def main() -> None: - ap = argparse.ArgumentParser(description=__doc__, - formatter_class=argparse.RawDescriptionHelpFormatter) - ap.add_argument("--norgate-store", required=True, - help="A Norgate-built store — a $MARKETDATA_STORE since ADR-0007.") - ap.add_argument("--symbols", nargs="+", default=_DEFAULT_SYMBOLS) - ap.add_argument("--rules", nargs="+", default=_DEFAULT_RULES, help="c=calendar n=OI v=volume") - ap.add_argument("--tol-days", nargs="+", type=int, default=[3], - help="Match window(s) in days. Pass several (e.g. 3 7 10) to sweep them " - "in one run — with the cache, extra tolerances cost no extra pulls.") - ap.add_argument("--dataset", default="GLBX.MDP3") - ap.add_argument("--cache-dir", default=".rollrule_cache", - help="Where to cache the pulled roll dates so re-runs and tolerance " - "sweeps skip the slow API pull (default ./.rollrule_cache).") - ap.add_argument("--refresh", action="store_true", help="Ignore the cache and re-pull.") - args = ap.parse_args() - - tols = sorted(set(args.tol_days)) - cache_dir = Path(args.cache_dir) - roots = {s.internal: s.databento for s in all_symbols()} - client = _client_from_env() - - winners = {} - for sym in args.symbols: - root = roots.get(sym) - if not root: - print(f"\n{sym}: no databento root in the registry (databento: null?) — skipping") - continue - ng = norgate_roll_dates(args.norgate_store, sym) - if len(ng) == 0: - print(f"\n{sym}: no Norgate roll dates found in {args.norgate_store} — skipping") - continue - print(f"\n{sym} (databento root {root!r}, Norgate rolls total {len(ng)})") - tol_hdr = " ".join(f"rate@{t}d" for t in tols) - print(f" {'rule':5} {'db_rolls':>8} {'ng_ovl':>7} {tol_hdr} {'med|off|d':>9}") - best = None # (rule, rate_at_widest_tol, med) — judged on the widest tolerance - for rule in args.rules: - try: - db, cached = roll_dates_cached(client, args.dataset, root, rule, cache_dir, args.refresh) - except Exception as e: # noqa: BLE001 — one bad rule should not sink the run - print(f" {rule:5} FETCH FAILED — {e}") - continue - rates = [score(db, ng, t) for t in tols] - n_db, n_ng = rates[0][0], rates[0][1] - rate_cells = " ".join(f"{r[3]:>6.3f}" for r in rates) - med = rates[-1][4] - tag = "" if not cached else " (cached)" - print(f" {rule:5} {n_db:>8} {n_ng:>7} {rate_cells} {med:>9.1f}{tag}") - wide = rates[-1][3] - if wide == wide and (best is None or wide > best[1] or (wide == best[1] and med < best[2])): - best = (rule, wide, med) - if best: - winners[sym] = best[0] - print(f" -> best match: rule '{best[0]}' (rate {best[1]:.3f} @ {tols[-1]}d, " - f"median offset {best[2]:.1f}d)") - - if winners: - print("\n=== recommended roll rule per symbol ===") - for sym, rule in winners.items(): - print(f" {sym:5} {rule}") - rules_used = set(winners.values()) - if len(rules_used) == 1: - print(f"\nAll tested symbols favor the same rule: '{rules_used.pop()}'. " - f"If it beats '.n' broadly, switch the producer's _FEEDS root.") - else: - print("\nDifferent symbols favor different rules — a single global rule may not fit; " - "consider a per-symbol roll-rule field in the registry.") - else: - print("\nNo symbols scored — check DATABENTO_API_KEY and the Norgate store path.") - sys.exit(1) - - -if __name__ == "__main__": - main() diff --git a/scripts/validate_databento_vs_norgate.py b/scripts/validate_databento_vs_norgate.py deleted file mode 100644 index 23b5c17..0000000 --- a/scripts/validate_databento_vs_norgate.py +++ /dev/null @@ -1,204 +0,0 @@ -#!/usr/bin/env python -"""ADR-0006 item 6: validate databento back-adjustment against Norgate. - -The databento and Norgate `backadj` series are built independently (single provider -per symbol, no stitching), so their absolute LEVELS may differ: different roll -calendars pick different roll dates, and the additive back-adjust anchor floats to -each provider's most-recent price. What must agree is the SHAPE. - -Additive (Panama) back-adjustment preserves absolute daily price CHANGES, not percent -returns — a floating anchor changes the price level and hence the return denominator, -but `Close.diff()` on any segment equals the true settlement change either way. So the -core check is: daily changes match between the two series, except near roll dates the -two providers place differently. This harness quantifies that. - -It reads `backadj` for a few liquid symbols from two stores — one built by Norgate, -one built by databento (`cotdata-update --build-databento`) — and reports, per symbol: - * overlap (date spans, common days), - * change correlation, scale ratio, and normalized worst-day change diff (the shape - check, unitless so one tolerance works across symbols), - * level difference stats (expected non-zero; informational), - * roll-date agreement (from Delivery Month changes). - -It exits non-zero if any symbol falls outside tolerance, so it can gate a promotion. -It needs real data, so it is NOT run in CI — the comparison logic is unit tested in -tests/test_validate_databento.py against synthetic frames. - -Since ADR-0007 the two stores belong to two packages: the Norgate side is a -$MARKETDATA_STORE (`bars/futures/norgate/`), the databento side is a $COTDATA_STORE -(`prices/`). `read_backadj` accepts either layout, so a Norgate store synced before -the move still reads. - -Usage: - python scripts/validate_databento_vs_norgate.py \ - --norgate-store /path/to/marketdata_store \ - --databento-store /path/to/cotdata_store \ - --symbols ES CL GC -""" -from __future__ import annotations - -import argparse -import sys -from pathlib import Path - -import pandas as pd - -_DEFAULT_SYMBOLS = ["ES", "CL", "GC"] -# Defaults are deliberately lenient — tighten once you have seen a clean run. A few -# days a year land on a mismatched roll and legitimately differ, so judge on the -# correlation and the bulk of days, not a single worst day. -_DEFAULT_CHANGE_CORR_MIN = 0.999 -_DEFAULT_SCALE_BAND = 0.02 # |scale_ratio - 1| allowed (catches unit/settlement scale bugs) -_DEFAULT_REL_TOL = 0.10 # worst |Δchange| as a fraction of a typical daily change -_DEFAULT_MAX_OUTLIER_DAYS = 8 # days/yr allowed to exceed REL_TOL (roll-date mismatches) - - -# Where a store keeps `_backadj.parquet`, newest layout first. -# ADR-0007 moved the Norgate side into `marketdata`, which keys a series by -# `bars///`; the databento side still writes cotdata's flat -# `prices/`. Both are read here rather than one, because this harness compares two -# stores that are now produced by two different packages — and because a Norgate -# store synced before the move still has the old shape. -_LAYOUTS = ("bars/futures/norgate", "prices") - - -def read_backadj(store_path: str, symbol: str) -> pd.DataFrame: - """Read one symbol's backadj OHLC frame straight from a store's parquet, - normalized to a tz-naive daily Date index. Empty if absent.""" - for layout in _LAYOUTS: - p = Path(store_path) / layout / f"{symbol}_backadj.parquet" - if p.exists(): - df = pd.read_parquet(p) - df.index = pd.to_datetime(df.index).tz_localize(None).normalize() - df.index.name = "Date" - return df.sort_index() - return pd.DataFrame() - - -def _roll_dates(df: pd.DataFrame) -> pd.DatetimeIndex: - """Roll dates = days the Delivery Month changes. Empty if the column is absent. - Values differ between providers (Norgate '202606' vs databento 'ESM6'), but the - DATES they change on are comparable.""" - if "Delivery Month" not in df.columns: - return pd.DatetimeIndex([]) - dm = df["Delivery Month"] - changed = dm.ne(dm.shift()) & dm.shift().notna() - return df.index[changed.fillna(False)] - - -def compare(norgate: pd.DataFrame, databento: pd.DataFrame, symbol: str = "?") -> dict: - """Compare two backadj frames over their date overlap. Returns a metrics dict.""" - idx = norgate.index.intersection(databento.index) - m: dict = {"symbol": symbol, "n_common": int(len(idx))} - if len(idx) < 3: - m["error"] = "insufficient overlap" - return m - - n = norgate.loc[idx, "Close"].astype(float) - d = databento.loc[idx, "Close"].astype(float) - - # Shape check on daily CHANGES (additive back-adjust preserves changes, not returns). - nc = n.diff().dropna() - dc = d.diff().reindex(nc.index) - scale = float(nc.abs().median()) or 1.0 # a typical daily change, for unitless norm - rel = (nc - dc).abs() / scale - m["change_corr"] = float(nc.corr(dc)) - m["scale_ratio"] = float(dc.std() / nc.std()) if float(nc.std()) else float("nan") - m["change_max_rel_diff"] = float(rel.max()) - m["change_rmse_points"] = float(((nc - dc) ** 2).mean() ** 0.5) - - # Level difference: informational (a piecewise-constant offset is expected). - lvl = (n - d).abs() - m["level_mean_abs"] = float(lvl.mean()) - m["level_max_abs"] = float(lvl.max()) - - # Coverage + roll agreement. - m["norgate_span"] = f"{n.index.min().date()}..{n.index.max().date()}" - m["databento_span"] = f"{d.index.min().date()}..{d.index.max().date()}" - nr_rolls = _roll_dates(norgate).intersection(idx) - db_rolls = _roll_dates(databento).intersection(idx) - m["rolls_norgate"] = int(len(nr_rolls)) - m["rolls_databento"] = int(len(db_rolls)) - m["rolls_common"] = int(len(nr_rolls.intersection(db_rolls))) - - years = max((idx.max() - idx.min()).days / 365.25, 1e-9) - m["outlier_days"] = int((rel > _DEFAULT_REL_TOL).sum()) - m["outlier_days_per_yr"] = m["outlier_days"] / years - return m - - -def evaluate(m: dict, corr_min: float, scale_band: float, rel_tol: float, - max_outliers_per_yr: float) -> list: - """Return a list of failure reasons for one symbol's metrics (empty = pass).""" - if "error" in m: - return [m["error"]] - fails = [] - if m["change_corr"] < corr_min: - fails.append(f"change_corr {m['change_corr']:.5f} < {corr_min}") - if abs(m["scale_ratio"] - 1.0) > scale_band: - fails.append(f"scale_ratio {m['scale_ratio']:.4f} off 1.0 by > {scale_band} " - f"(unit/settlement scale mismatch?)") - # A big worst-day diff is only a failure if it happens on too many days (roll noise - # on a handful of days a year is expected and fine). - n_over = round(m["outlier_days_per_yr"], 1) - if m["change_max_rel_diff"] > rel_tol and m["outlier_days_per_yr"] > max_outliers_per_yr: - fails.append(f"{n_over} days/yr exceed rel tol {rel_tol} " - f"(worst {m['change_max_rel_diff']:.2f}x a typical day)") - return fails - - -def format_report(m: dict, fails: list) -> str: - if "error" in m: - return f" {m['symbol']:<5} SKIP ({m['error']}, n_common={m['n_common']})" - status = "PASS" if not fails else "FAIL" - lines = [ - f" {m['symbol']:<5} {status} common={m['n_common']} " - f"norgate={m['norgate_span']} databento={m['databento_span']}", - f" change_corr={m['change_corr']:.5f} scale_ratio={m['scale_ratio']:.4f} " - f"worst_rel={m['change_max_rel_diff']:.2f} rmse={m['change_rmse_points']:.4f}pts " - f"outliers={m['outlier_days']} ({m['outlier_days_per_yr']:.1f}/yr)", - f" level|Δ| mean={m['level_mean_abs']:.4f} max={m['level_max_abs']:.4f} " - f"rolls n/db/common={m['rolls_norgate']}/{m['rolls_databento']}/{m['rolls_common']}", - ] - if fails: - lines.append(" FAIL: " + "; ".join(fails)) - return "\n".join(lines) - - -def main(argv=None) -> int: - p = argparse.ArgumentParser(description="Validate databento backadj against Norgate.") - p.add_argument("--norgate-store", required=True, - help="Path to the Norgate-built store — a $MARKETDATA_STORE since " - "ADR-0007. The pre-move cotdata layout is still accepted.") - p.add_argument("--databento-store", required=True, help="Path to the databento-built store.") - p.add_argument("--symbols", nargs="+", default=_DEFAULT_SYMBOLS) - p.add_argument("--change-corr-min", type=float, default=_DEFAULT_CHANGE_CORR_MIN) - p.add_argument("--scale-band", type=float, default=_DEFAULT_SCALE_BAND) - p.add_argument("--rel-tol", type=float, default=_DEFAULT_REL_TOL) - p.add_argument("--max-outlier-days-per-yr", type=float, default=_DEFAULT_MAX_OUTLIER_DAYS) - args = p.parse_args(argv) - - print(f"Validating databento vs Norgate backadj on: {', '.join(args.symbols)}") - any_fail, any_data = False, False - for sym in args.symbols: - ng = read_backadj(args.norgate_store, sym) - db = read_backadj(args.databento_store, sym) - if ng.empty or db.empty: - missing = ", ".join(s for s, df in (("norgate", ng), ("databento", db)) if df.empty) - print(f" {sym:<5} SKIP (no backadj in: {missing})") - continue - any_data = True - m = compare(ng, db, sym) - fails = evaluate(m, args.change_corr_min, args.scale_band, args.rel_tol, - args.max_outlier_days_per_yr) - any_fail = any_fail or bool(fails) - print(format_report(m, fails)) - - if not any_data: - print("No comparable symbols found in both stores.") - return 2 - return 1 if any_fail else 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/src/cotdata/__init__.py b/src/cotdata/__init__.py index 1838fa0..275ec5e 100644 --- a/src/cotdata/__init__.py +++ b/src/cotdata/__init__.py @@ -1,15 +1,16 @@ """cotdata — canonical CFTC positioning layer (see README). Public consumer API. Bars are NOT here. ADR-0007 makes this package COT-only and moves every price -series to `marketdata`, so `get_prices`/`roll_dates` are gone — import -``marketdata.get_bars`` instead. The two packages keep separate stores -($COTDATA_STORE, $MARKETDATA_STORE) and separate producers. +series to `marketdata` — Norgate, databento and Yahoo alike — so `get_prices` and +`roll_dates` are gone; import ``marketdata.get_bars`` instead. The two packages +keep separate stores ($COTDATA_STORE, $MARKETDATA_STORE), separate registries and +separate producers. """ from .cot import get_cot from .registry import REGISTRY, Symbol, all_symbols, symbol from .store import load_manifest, require_schema, schema_version -__version__ = "0.4.0" +__version__ = "0.5.0" __all__ = [ "get_cot", "symbol", "all_symbols", "REGISTRY", "Symbol", diff --git a/src/cotdata/config.py b/src/cotdata/config.py index 6d3f1ed..941c54a 100644 --- a/src/cotdata/config.py +++ b/src/cotdata/config.py @@ -21,6 +21,12 @@ def store_root() -> Path: return Path(root) +# LEGACY, read-only. Nothing writes either directory any more — bars, contract specs +# and the databento producer all moved to marketdata (ADR-0007). They survive because +# `store.reconcile_manifest` resolves a domain's directory through `_DOMAIN_DIRS`, and +# a store written before those moves still carries `prices` and `metadata` entries to +# reconcile. Deleting these would make reconcile fall back to a guessed path for +# exactly the entries it exists to clean up. def prices_dir() -> Path: return store_root() / "prices" diff --git a/src/cotdata/providers/databento.py b/src/cotdata/providers/databento.py deleted file mode 100644 index 1ba7069..0000000 --- a/src/cotdata/providers/databento.py +++ /dev/null @@ -1,1099 +0,0 @@ -"""Databento price provider for cotdata (GLBX.MDP3). Two parts live in this file. - -LIVE, the ADR-0006 two-stage producer (an alternative to Norgate, selected per deployment -via COTDATA_PRICE_SOURCE): - * ingest() fetches raw .n.0/.n.1 ohlcv-1d + statistics into an append-only raw bronze - store ($COTDATA_DATABENTO_RAW). The only paid step, resumable, paged by year. - * build() reads only the raw store and writes back-adjusted prices to the cotdata store. - Free and re-runnable. -Driven by `cotdata-update --ingest-databento` / `--build-databento`. databento is a -validated ALTERNATIVE provider (ADR-0006 Accepted), producing provider-different series -from Norgate, so it is not the default source. See crucible-stack ADR-0006 and -docs/databento_norgate_parity.md. - -DORMANT, an older per-symbol EOD path (fetch_daily_ohlc, run_batch_backfill, -update_all_daily_prices) that is on NO live path. Kept deliberately for (1) the intraday -news-failure work (Norgate has no intraday, so databento would source the release-window -reaction refinement of the CMR trigger) and (2) cross-checking Norgate's settlement close -against databento statistics. Remove it if that intraday work is abandoned. - -The hard-won piece both parts share is the STATISTICS extraction: Open Interest is -stat_type 9, and settlement is StatType.SETTLEMENT_PRICE == 3 (NOT 7 = LOWEST_OFFER, which -overwrote Close with the day's lowest offer), dated by ts_ref (the session it applies to), -not ts_event (the final settle is disseminated the next morning). - -Lazy `import databento` (behind the [databento] extra), symbols from the registry, -requires DATABENTO_API_KEY for the paid fetches. -""" -import datetime as dt -import json -import logging -import os -import sys -import time -import warnings -from pathlib import Path -from typing import Optional - -import pandas as pd - -from .. import config, store -from ..registry import all_symbols - -logger = logging.getLogger(__name__) - -# Symbols Databento GLBX.MDP3 doesn't carry as a .n.0 continuous (fall back to yfinance). -# Derived from the registry (`databento: null`), the authoritative capability mapping, so -# it can't drift from it the way the old hardcoded list did (it missed DX/MME/MFS). -_DATABENTO_UNSUPPORTED = frozenset(s.internal for s in all_symbols() if s.databento is None) -_API_LAST_CHECKED = {} # in-memory throttle to avoid hammering the API - - -def _cache_dir() -> Path: - d = config.store_root() / "_cache" / "databento" - d.mkdir(parents=True, exist_ok=True) - return d - - -def run_batch_backfill(symbols: list) -> None: - """Submit a Databento batch job to download massive historical daily data.""" - import databento as db - db_key = os.environ.get("DATABENTO_API_KEY") - if not db_key: - raise ValueError("DATABENTO_API_KEY is missing. Cannot run batch backfill.") - client = db.Historical(key=db_key) - - db_syms, sym_map = [], {} - for sym in symbols: - db_sym = f"{sym}.n.0" - db_syms.append(db_sym) - sym_map[db_sym] = sym - - try: - res = client.symbology.resolve( - dataset="GLBX.MDP3", symbols=db_syms, stype_in="continuous", - stype_out="instrument_id", start_date="2010-06-06", - end_date=pd.Timestamp.now().strftime("%Y-%m-%d"), - ) - not_found = res.get("not_found", []) - if not_found: - db_syms = [s for s in db_syms if s not in not_found] - except Exception as e: # noqa: BLE001 - logger.warning("Symbol resolution failed, submitting anyway: %s", e) - if not db_syms: - raise ValueError("No valid Databento symbols remain after validation.") - - job = client.batch.submit_job( - dataset="GLBX.MDP3", symbols=db_syms, stype_in="continuous", schema="ohlcv-1d", - start="2010-06-06", end=pd.Timestamp.now().strftime("%Y-%m-%d"), - encoding="csv", split_symbols=False, delivery="download", - ) - job_id = job.get("id") - if not job_id: - raise RuntimeError("Failed to submit batch job: no job ID returned.") - - while True: - time.sleep(30) - my_job = next((j for j in client.batch.list_jobs() if j.get("id") == job_id), None) - if not my_job: - continue - state = my_job.get("state", "unknown") - if state == "done": - break - if state == "expired" or "fail" in state.lower(): - raise RuntimeError(f"Batch job failed or expired: {my_job}") - - import tempfile - with tempfile.TemporaryDirectory() as tmpdir: - paths = client.batch.download(job_id=job_id, output_dir=tmpdir) - for path in paths: - if not str(path).endswith(".csv"): - continue - df_raw = pd.read_csv(path) - if "symbol" not in df_raw.columns: - continue - for db_sym, df_sym in df_raw.groupby("symbol"): - internal_sym = sym_map.get(db_sym) - if not internal_sym: - continue - df_sym = df_sym.rename(columns={ - "ts_event": "Date", "open": "Open", "high": "High", - "low": "Low", "close": "Close", "volume": "Volume"}) - df_sym["Date"] = pd.to_datetime(df_sym["Date"]).dt.tz_localize(None).dt.normalize() - df_sym = df_sym.set_index("Date") - keep = [c for c in ("Open", "High", "Low", "Close", "Volume") if c in df_sym.columns] - df_clean = df_sym[keep].copy() - df_clean["Open Interest"] = float("nan") - df_clean = df_clean.dropna(subset=["Close"]).sort_index() - df_clean.to_parquet(_cache_dir() / f"{internal_sym}_daily.parquet") - - -def fetch_daily_ohlc(symbol: str, start_date: Optional[str] = None, - force_refresh: bool = False, price_type: str = "close") -> pd.DataFrame: - """Daily OHLC + Open Interest via Databento (GLBX.MDP3 ohlcv-1d + statistics), - append-only cache; yfinance fallback for _DATABENTO_UNSUPPORTED. price_type - 'settlement' pulls stat_type 3 dated by ts_ref. - - The cache is always a single, from-inception, append-only series per - symbol+price_type, shared across every caller — that is what makes repeated - calls cheap. `start_date` therefore means two different things depending on - whether it is the FIRST call for a symbol or not, and both are deliberate: - - * **Cold cache (symbol never fetched before):** `start_date` clamps the fetch - floor (still no lower than the GLBX.MDP3 history floor 2010-06-06), so a - narrow first-time query — e.g. "just the last 3 months" — actually costs a - narrow API pull, not a from-2000 one. - * **Warm cache (symbol already has some history):** `start_date` does NOT - change what gets fetched — the top-up always resumes from the cache's - `last_date + 1 day`, exactly as before. Letting a later, narrower - `start_date` shrink the fetch would silently truncate a cache other - callers already rely on being complete. `start_date` still filters what is - RETURNED to this call, just not what is fetched or persisted. - - In both cases the returned frame never contains rows before `start_date` — the - filter is applied uniformly to whatever the cache ends up holding. `None` - (the default) is unbounded: full cached history, unfiltered, unchanged - behaviour from before this parameter existed. - - One consequence of the "warm cache never shrinks its floor" rule: if a symbol's - cache was FIRST populated by a narrow `start_date` (e.g. "since 2024"), a later - call asking for `start_date="2010-01-01"` will NOT backfill 2010-2023 — the - cache only ever grows forward. Not a concern for the one real caller today - (`update_all_daily_prices`, which never passes `start_date` and always wants - full history), but worth knowing before relying on this for a symbol multiple - call sites touch with different windows. - """ - import databento as db - display_sym = symbol - _pt_suffix = "" if price_type == "close" else f"_{price_type}" - cache_path = _cache_dir() / f"{symbol}_daily{_pt_suffix}.parquet" - - local_df = pd.DataFrame() - if cache_path.exists() and not force_refresh: - try: - cached = pd.read_parquet(cache_path) - if not cached.empty: - has_oi = "Open Interest" in cached.columns and not cached["Open Interest"].isna().all() - if has_oi or symbol in _DATABENTO_UNSUPPORTED: - local_df = cached - except Exception as e: # noqa: BLE001 - logger.warning("Failed to read cache for %s: %s", display_sym, e) - - def _filtered(df: pd.DataFrame) -> pd.DataFrame: - if start_date and not df.empty: - return df[df.index >= pd.Timestamp(start_date)] - return df - - if local_df.empty: - # Cold cache: start_date narrows the fetch floor (still API-cost-bounded - # by the GLBX floor below), so a first-time narrow query is actually cheap. - fetch_start = start_date if start_date else "2000-01-01" - else: - # Warm cache: start_date does NOT narrow this — see the docstring. It only - # affects the return-value filter below. - last_date = local_df.index.max() - today = pd.Timestamp.now().normalize() - if last_date >= today - pd.Timedelta(days=1): - return _filtered(local_df) - if "--fast" in sys.argv: - return _filtered(local_df) - now = time.time() - if not force_refresh and (now - _API_LAST_CHECKED.get(symbol, 0) < 3600): - return _filtered(local_df) - _API_LAST_CHECKED[symbol] = now - fetch_start = (last_date + pd.Timedelta(days=1)).strftime("%Y-%m-%d") - - fetch_end = (pd.Timestamp.now() - pd.Timedelta(days=1)).strftime("%Y-%m-%d") - if pd.Timestamp(fetch_start) >= pd.Timestamp(fetch_end): - return _filtered(local_df) - - new_df = pd.DataFrame() - db_key = os.environ.get("DATABENTO_API_KEY") - if pd.Timestamp(fetch_start) < pd.Timestamp("2010-06-06"): - fetch_start = "2010-06-06" # GLBX.MDP3 history floor - db_success = False - - if db_key and symbol not in _DATABENTO_UNSUPPORTED: - for attempt in range(1, 4): - try: - client = db.Historical(key=db_key) - db_sym = f"{symbol}.n.0" - with warnings.catch_warnings(): - warnings.filterwarnings("ignore", message=".*No data found.*") - warnings.filterwarnings("ignore", message=".*did not resolve.*") - data = client.timeseries.get_range( - dataset="GLBX.MDP3", symbols=[db_sym], stype_in="continuous", - schema="ohlcv-1d", start=fetch_start, end=fetch_end) - raw = data.to_df() - if not raw.empty: - raw = raw.reset_index().rename(columns={ - "ts_event": "Date", "open": "Open", "high": "High", - "low": "Low", "close": "Close", "volume": "Volume"}) - raw["Date"] = pd.to_datetime(raw["Date"]).dt.tz_localize(None).dt.normalize() - raw = raw.set_index("Date") - keep = [c for c in ("Open", "High", "Low", "Close", "Volume") if c in raw.columns] - new_df = raw[keep].copy() - try: - with warnings.catch_warnings(): - warnings.filterwarnings("ignore", message=".*No data found.*") - warnings.filterwarnings("ignore", message=".*did not resolve.*") - stat_raw = client.timeseries.get_range( - dataset="GLBX.MDP3", symbols=[db_sym], stype_in="continuous", - schema="statistics", start=fetch_start, end=fetch_end).to_df() - if not stat_raw.empty: - stat_raw = stat_raw.reset_index() - if "stat_type" in stat_raw.columns: - new_df = new_df.reset_index() - date_col = "ts_event" - # Open Interest = stat_type 9 - oi_df = stat_raw[stat_raw["stat_type"] == 9].copy() - if not oi_df.empty: - oi_df["Date"] = pd.to_datetime(oi_df[date_col]).dt.tz_localize(None).dt.normalize() - oi_df["Open Interest"] = oi_df["quantity"] if "quantity" in oi_df.columns else oi_df["price"] - oi_df = oi_df.groupby("Date")["Open Interest"].last().reset_index() - new_df = pd.merge(new_df, oi_df, on="Date", how="left") - # Settlement = stat_type 3 (NOT 7=LOWEST_OFFER), dated by ts_ref - if price_type == "settlement": - stl_df = stat_raw[stat_raw["stat_type"] == 3].copy() - if not stl_df.empty: - settle_dt = "ts_ref" if "ts_ref" in stl_df.columns else date_col - stl_df["Date"] = pd.to_datetime(stl_df[settle_dt]).dt.tz_localize(None).dt.normalize() - stl_df["Settlement"] = stl_df["price"] - stl_df = stl_df.groupby("Date")["Settlement"].last().reset_index() - new_df = pd.merge(new_df, stl_df, on="Date", how="left") - if "Settlement" in new_df.columns: - new_df["Close"] = new_df["Settlement"].combine_first(new_df["Close"]) - new_df = new_df.drop(columns=["Settlement"]) - new_df = new_df.set_index("Date") - except Exception as e: # noqa: BLE001 - logger.warning("Could not fetch Open Interest for %s: %s", display_sym, e) - if "Open Interest" not in new_df.columns: - new_df["Open Interest"] = float("nan") - db_success = True - break - db_success = True # empty (holiday) — don't trigger fallback - break - except Exception as exc: # noqa: BLE001 - logger.warning("Databento download failed for %s (%d/3): %s", display_sym, attempt, exc) - if attempt < 3: - time.sleep(5) - - if not db_success and symbol in _DATABENTO_UNSUPPORTED: - try: - import yfinance as yf - yf_df = yf.download(f"{symbol}=F", start=fetch_start, - end=pd.Timestamp.now().strftime("%Y-%m-%d"), progress=False) - if not yf_df.empty: - yf_df = yf_df.reset_index() - if isinstance(yf_df.columns, pd.MultiIndex): - yf_df.columns = [c[0] for c in yf_df.columns] - yf_df["Date"] = pd.to_datetime(yf_df["Date"]).dt.tz_localize(None).dt.normalize() - yf_df = yf_df.set_index("Date") - keep = [c for c in ("Open", "High", "Low", "Close", "Volume") if c in yf_df.columns] - new_df = yf_df[keep].copy() - new_df["Open Interest"] = float("nan") - except Exception as yf_exc: # noqa: BLE001 - logger.error("yfinance fallback failed for %s: %s", display_sym, yf_exc) - - if not local_df.empty and not new_df.empty: - new_df = new_df[new_df.index > local_df.index.max()] - if new_df.empty: - return _filtered(local_df) - new_df = new_df.dropna(subset=["Close"]) - combined = pd.concat([local_df, new_df]) if not local_df.empty else new_df - combined = combined[~combined.index.duplicated(keep="last")].sort_index() - combined.to_parquet(cache_path) # cache always persists the FULL series - return _filtered(combined) # start_date only shapes what's returned - - -def update_all_daily_prices(force_refresh: bool = False) -> None: - """Refresh the Databento price cache for all registry symbols (dormant path).""" - to_batch = [] - for s in all_symbols(): - symbol = s.internal - raw_cache_path = _cache_dir() / f"{symbol}_daily.parquet" - needs = force_refresh or not raw_cache_path.exists() - if not needs: - try: - if pd.read_parquet(raw_cache_path).empty: - needs = True - except Exception: # noqa: BLE001 - needs = True - if needs: - to_batch.append(symbol) - else: - try: - fetch_daily_ohlc(symbol) - except Exception as e: # noqa: BLE001 - logger.error("Failed daily gap fetch for %s: %s", symbol, e) - if to_batch: - try: - run_batch_backfill(to_batch) - except Exception as e: # noqa: BLE001 - logger.error("Batch backfill failed: %s", e) - - -# ── Raw ingest (Stage 1): the paid, append-only landing store ──────────────── -# ADR-0006: databento is a two-stage producer. Stage 1 (here) is the ONLY step -# that hits the paid API. It pulls raw .n.0 / .n.1 ohlcv-1d + statistics into an -# immutable, append-only raw store, keyed by fetched date range in a manifest so a -# re-run resumes from last_date+1 and never re-pulls a range already held. The free -# Stage 2 `build` (see ADR item 4) re-derives back-adjusted prices from these local -# files with no API cost. The raw store is PRODUCER-INTERNAL, not the consumer -# contract — keep it out of any store sync to consumers. - -GLBX_HISTORY_FLOOR = "2010-06-06" # earliest GLBX.MDP3 history -_FEEDS = (".n.0", ".n.1") # front + second continuous (second gives the roll gap) -_SCHEMAS = ("ohlcv-1d", "statistics") - - -def raw_root() -> Path: - """Producer-internal databento raw store: $COTDATA_DATABENTO_RAW if set, else a - ``_raw/databento`` namespace under the cotdata store (leading underscore = not a - consumer domain; exclude it from any consumer sync).""" - env = os.environ.get("COTDATA_DATABENTO_RAW", "").strip() - return Path(env) if env else (config.store_root() / "_raw" / "databento") - - -def _raw_path(symbol: str, feed: str, schema: str) -> Path: - sub = "ohlcv" if schema == "ohlcv-1d" else "statistics" - return raw_root() / sub / f"{symbol}{feed}.parquet" - - -def _ingest_manifest_path() -> Path: - return raw_root() / "ingest_manifest.json" - - -def _load_ingest_manifest() -> dict: - p = _ingest_manifest_path() - return json.loads(p.read_text()) if p.exists() else {} - - -def _write_ingest_manifest(m: dict) -> None: - p = _ingest_manifest_path() - p.parent.mkdir(parents=True, exist_ok=True) - tmp = p.with_suffix(".json.tmp") - tmp.write_text(json.dumps(m, indent=2, sort_keys=True)) - os.replace(tmp, p) - - -def _to_naive(x): - """tz-naive UTC for either tz-aware or naive datetime input (databento returns UTC). - Handles both a DatetimeIndex (``.tz_convert``) and a Series (``.dt.tz_convert``).""" - ts = pd.to_datetime(x, utc=True) - return ts.dt.tz_convert(None) if isinstance(ts, pd.Series) else ts.tz_convert(None) - - -def _normalize(raw: pd.DataFrame, schema: str) -> pd.DataFrame: - """Light bronze normalization: tz-naive timestamps, one row per day for ohlcv. - Columns are otherwise preserved as databento returns them, so Stage 2 can - re-extract settlement/OI/etc. without a re-fetch.""" - raw = raw.copy() - if schema == "ohlcv-1d": - raw.index = _to_naive(raw.index).normalize() - raw.index.name = "Date" - return raw[~raw.index.duplicated(keep="last")].sort_index() - # statistics: flatten ts_event out of the index, keep every stat row. - raw = raw.reset_index() - for c in ("ts_event", "ts_ref"): - if c in raw.columns: - raw[c] = _to_naive(raw[c]) - return raw.drop_duplicates().reset_index(drop=True) - - -def _date_bounds(raw: pd.DataFrame, schema: str): - if schema == "ohlcv-1d": - return str(raw.index.min().date()), str(raw.index.max().date()) - if "ts_event" in raw.columns and len(raw): - d = pd.to_datetime(raw["ts_event"]) - return str(d.min().date()), str(d.max().date()) - return None, None - - -def _append_raw(symbol: str, feed: str, schema: str, new_df: pd.DataFrame) -> None: - path = _raw_path(symbol, feed, schema) - existing = pd.read_parquet(path) if path.exists() else pd.DataFrame() - combined = pd.concat([existing, new_df]) if not existing.empty else new_df - if schema == "ohlcv-1d": - combined = combined[~combined.index.duplicated(keep="last")].sort_index() - else: - combined = combined.drop_duplicates().reset_index(drop=True) - store._atomic_write_parquet(combined, path) - - -def _client_from_env(): - import databento as db # lazy — behind the [databento] extra - key = os.environ.get("DATABENTO_API_KEY") - if not key: - raise RuntimeError("DATABENTO_API_KEY is not set; cannot ingest from databento.") - return db.Historical(key=key) - - -def _fetch(client, dataset: str, dbsym: str, schema: str, start: str, end: str, - *, retries: int = 3, backoff: float = 5.0) -> pd.DataFrame: - """One databento get_range, with retry + linear backoff on transient network - failures (read timeouts, dropped/aborted streams) — the historical API throws these - often on large pulls. Raises the last error only after ``retries`` attempts.""" - last = None - for attempt in range(1, retries + 1): - try: - with warnings.catch_warnings(): - warnings.filterwarnings("ignore", message=".*No data found.*") - warnings.filterwarnings("ignore", message=".*did not resolve.*") - # Databento flags a handful of historically "degraded" sessions (e.g. - # 2014-06-11) on every request. A known data-condition note, not an error, - # and it floods the logs — quiet it. - warnings.filterwarnings("ignore", message=".*reduced quality.*") - data = client.timeseries.get_range( - dataset=dataset, symbols=[dbsym], stype_in="continuous", - schema=schema, start=start, end=end) - return data.to_df() - except Exception as e: # noqa: BLE001 — databento/network is flaky; retry transient - last = e - if attempt < retries: - time.sleep(backoff * attempt) - raise last - - -def _fmt_hms(seconds: float) -> str: - """Compact H:MM:SS for progress logging.""" - s = int(seconds) - return f"{s // 3600}:{(s % 3600) // 60:02d}:{s % 60:02d}" - - -# A from-inception statistics pull in a single get_range times out (the streaming read -# runs for many minutes and the server drops it). Page it into calendar-year windows so -# each request is small and completes, and the manifest advances per window. -_STATS_CHUNK_DAYS = 365 - - -def _stream_into_raw(client, dataset, internal, dbsym, feed, schema, start, end, - manifest, key, *, chunk_days=None): - """Stream ``[start, end)`` into the raw store, paging by ``chunk_days`` (None = one - request). Appends and persists the manifest after EACH page, so a from-inception pull - can't time out in one giant request and a mid-history failure resumes at the last good - page instead of from scratch. Returns ``(rows_added, completed_ok)``.""" - end_ts = pd.Timestamp(end) - cur = pd.Timestamp(start) - added = 0 - while cur < end_ts: - c_end = min(cur + pd.Timedelta(days=chunk_days), end_ts) if chunk_days else end_ts - t0 = time.monotonic() - try: - raw = _fetch(client, dataset, dbsym, schema, - cur.strftime("%Y-%m-%d"), c_end.strftime("%Y-%m-%d")) - except Exception as e: # noqa: BLE001 — databento/network is flaky - print(f" {internal}{feed} {schema}: fetch failed " - f"({cur.date()}..{c_end.date()}) after {time.monotonic() - t0:.1f}s — {e}") - return added, False - raw = _normalize(raw, schema) if raw is not None and not raw.empty else raw - rec = manifest.get(key, {}) - if raw is not None and not raw.empty: - _append_raw(internal, feed, schema, raw) - first, newest = _date_bounds(raw, schema) - manifest[key] = { - "first_date": rec.get("first_date") or first, - "last_date": newest, - "n_rows": int(rec.get("n_rows", 0)) + len(raw), - "fetched_at": dt.datetime.utcnow().isoformat(timespec="seconds") + "Z", - } - _write_ingest_manifest(manifest) - added += len(raw) - print(f" {internal}{feed} {schema}: +{len(raw):>8,} rows " - f"({cur.date()}..{newest}) in {time.monotonic() - t0:.1f}s") - elif chunk_days: - # An empty page inside a paged pull — a symbol whose data starts after the GLBX - # floor. Advance past it so a re-run doesn't refetch the empty span forever; - # only when paging, so a single empty one-shot stays a plain "nothing new" no-op. - manifest[key] = {**rec, - "last_date": (c_end - pd.Timedelta(days=1)).strftime("%Y-%m-%d")} - _write_ingest_manifest(manifest) - cur = c_end - return added, True - - -def ingest(symbols=None, *, client=None, dataset="GLBX.MDP3", end=None, - cold_start=GLBX_HISTORY_FLOOR, n1_stats_window=None) -> dict: - """Fetch raw databento daily bars (.n.0 + .n.1) and statistics into the raw - store, append-only. Resumes each (symbol, feed, schema) from its manifest - last_date+1 — the manifest is persisted after EVERY fetch, so an interrupted run - (a from-inception ``statistics`` pull can take minutes) resumes where it left off - instead of re-downloading. Logs per-asset progress ``[i/N]`` and timing. Scoped to - registry symbols that databento can serve (a non-null ``databento`` mapping); pass - ``symbols`` to narrow further. - - ``n1_stats_window`` (days): when set, the n.1 ``statistics`` schema is fetched only in - ±window-day windows around each roll date (found from the on-disk n.0 ohlcv) instead - of its full history. n.1 settlement is only used at rolls, so this drops the biggest - avoidable download with NO back-adjustment accuracy loss. The full n.0 statistics - (daily settlement + OI) and n.1 ohlcv are unaffected. - - `client` is injectable (databento.Historical-shaped) for tests; the default - builds one from DATABENTO_API_KEY. Returns {kind, ok, symbols, rows}.""" - targets = [s for s in all_symbols() - if s.databento and s.price_source in (None, "databento") - and (symbols is None or s.internal in symbols)] - if not targets: - print("databento ingest: no databento-capable symbols" - + (f" among {symbols}" if symbols else "")) - return {"kind": "ingest_databento", "ok": True, "symbols": 0, "rows": 0} - - if client is None: - client = _client_from_env() - - end = end or (pd.Timestamp.now().normalize() - pd.Timedelta(days=1)).strftime("%Y-%m-%d") - manifest = _load_ingest_manifest() - total_rows, failed = 0, 0 - n = len(targets) - run_start = time.monotonic() - - for i, s in enumerate(targets, 1): - sym_start = time.monotonic() - sym_rows = 0 - print(f"[{i}/{n}] {s.internal}: ingesting (run elapsed {_fmt_hms(time.monotonic() - run_start)})") - for feed in _FEEDS: - dbsym = f"{s.databento}{feed}" - for schema in _SCHEMAS: - if (n1_stats_window is not None and feed == ".n.1" - and schema == "statistics"): - added = _ingest_n1_stats_windowed( - client, dataset, s, manifest, end, n1_stats_window) - sym_rows += added - total_rows += added - continue - key = f"{s.internal}{feed}:{schema}" - rec = manifest.get(key, {}) - last = rec.get("last_date") - start = ((pd.Timestamp(last) + pd.Timedelta(days=1)).strftime("%Y-%m-%d") - if last else cold_start) - if pd.Timestamp(start) < pd.Timestamp(GLBX_HISTORY_FLOOR): - start = GLBX_HISTORY_FLOOR - # Already current. databento requires start < end; a start == end query - # 422s (data_time_range_start_on_or_after_end), so skip on >=. - if pd.Timestamp(start) >= pd.Timestamp(end): - continue - # Page the streaming fetch: a from-inception statistics pull in one giant - # get_range times out, so statistics is fetched in yearly windows with the - # manifest advanced per window. ohlcv is ~250 rows/yr, so one request is - # fine. A page failure leaves the manifest at the last good page to resume. - chunk = _STATS_CHUNK_DAYS if schema == "statistics" else None - added, completed = _stream_into_raw( - client, dataset, s.internal, dbsym, feed, schema, - start, end, manifest, key, chunk_days=chunk) - sym_rows += added - total_rows += added - if not completed: - failed += 1 - print(f"[{i}/{n}] {s.internal}: done in {_fmt_hms(time.monotonic() - sym_start)} " - f"({sym_rows:,} rows)") - - _write_ingest_manifest(manifest) - print(f"databento ingest: {n} symbols, {total_rows:,} rows in " - f"{_fmt_hms(time.monotonic() - run_start)}" - + (f" ({failed} fetch failure(s))" if failed else "")) - return {"kind": "ingest_databento", "ok": failed == 0, "symbols": n, "rows": total_rows} - - -def _roll_dates_on_disk(symbol: str) -> pd.DatetimeIndex: - """Roll dates for a symbol, read from the raw n.0 ohlcv already on disk (the last - session before the front contract's ``instrument_id`` changes). Empty if n.0 ohlcv is - absent — the windowed n.1 stats fetch needs n.0 ohlcv ingested first (it always is, - since .n.0/ohlcv-1d is fetched before .n.1/statistics in the loop).""" - n0 = _read_ohlcv(symbol, ".n.0") - if n0.empty: - return pd.DatetimeIndex([]) - key = _roll_key(n0) - if key is None: - return pd.DatetimeIndex([]) - ids = n0[key] - is_roll = ids.ne(ids.shift(-1)) & ids.shift(-1).notna() - return n0.index[is_roll] - - -def _ingest_n1_stats_windowed(client, dataset, s, manifest, end, window) -> int: - """Fetch n.1 ``statistics`` only in ±``window``-day windows around each roll date, - instead of the full history. Settlement is only read at rolls, so this is accuracy- - neutral. Resumes from the newest roll already covered. Returns rows added.""" - key = f"{s.internal}.n.1:statistics" - rolls = _roll_dates_on_disk(s.internal) - if len(rolls) == 0: - print(f" {s.internal}.n.1 statistics: no roll dates yet (n.0 ohlcv missing) — skipped") - return 0 - rec = manifest.get(key, {}) - covered = pd.Timestamp(rec["last_date"]) if rec.get("last_date") else None - todo = [d for d in rolls if covered is None or d > covered] - if not todo: - return 0 - dbsym = f"{s.databento}.n.1" - end_ts = pd.Timestamp(end) - added, t0 = 0, time.monotonic() - # Advance the resume watermark only through the last CONTIGUOUS success, so a transient - # mid-history window failure is retried on the next run rather than silently leaving that - # roll's gap unmeasured. - watermark, ok = covered, True - for d in todo: - wstart = d - pd.Timedelta(days=window) - wend = min(d + pd.Timedelta(days=window + 1), end_ts) # exclusive end - if wstart >= wend: - continue - try: - raw = _fetch(client, dataset, dbsym, "statistics", - wstart.strftime("%Y-%m-%d"), wend.strftime("%Y-%m-%d")) - except Exception as e: # noqa: BLE001 — databento/network is flaky - print(f" {s.internal}.n.1 statistics [{d.date()}]: fetch failed — {e}") - ok = False - continue - if raw is not None and not raw.empty: - raw = _normalize(raw, "statistics") - if not raw.empty: - _append_raw(s.internal, ".n.1", "statistics", raw) - added += len(raw) - if ok: - watermark = d - if watermark is not None: - manifest[key] = { - "first_date": rec.get("first_date") or str(todo[0].date()), - "last_date": str(pd.Timestamp(watermark).date()), - "n_rows": int(rec.get("n_rows", 0)) + added, - "fetched_at": dt.datetime.utcnow().isoformat(timespec="seconds") + "Z", - "windowed": True, - } - _write_ingest_manifest(manifest) - print(f" {s.internal}.n.1 statistics: +{added:>8,} rows across {len(todo)} roll " - f"window(s) (±{window}d) in {time.monotonic() - t0:.1f}s" - + ("" if ok else " (some windows failed — retry on re-run)")) - return added - - -def _claims_rows(entry) -> bool: - """Does this manifest entry assert that data was actually fetched? - - The ingest manifest holds two shapes, and only one of them implies a file: - - * a **record** of a fetched table, carrying `first_date` and `n_rows` - * an **advance marker**, `{last_date: ...}` and nothing else, written when a - paged pull meets an empty window so a re-run skips that span instead of - refetching it forever - - Only a record can be a ghost. Treated as a record when it claims rows, so an - entry that is malformed or from a future writer errs toward being KEPT: a - wrongly-kept entry costs one skipped table that `--ingest-databento` reports, - while a wrongly-pruned marker costs a paid refetch on every run thereafter. - """ - if not isinstance(entry, dict): - return False - try: - return int(entry.get("n_rows") or 0) > 0 - except (TypeError, ValueError): - return False - - -def reconcile_manifest(*, prune: bool = True) -> dict: - """Make the ingest manifest match the raw parquet files actually on disk. - - The manifest is the resume ledger: ingest() computes each table's start date from - ``last_date`` and skips the table entirely when that is already current. It never - checks that the file exists. So the manifest drifting out of step with the disk is - silently destructive in BOTH directions, and this fixes both. - - * **Manifest behind disk.** ingest() writes raw parquets incrementally, so a run - interrupted before it persisted the manifest leaves fetched tables unrecorded and - a restart re-downloads them. Backfilled from the files. - * **Manifest ahead of disk.** An entry that claims rows but whose parquet is missing - (a partial copy, a store move, a deleted file) still carries a current - ``last_date``, so a restart marks it "already current" and NEVER fetches it. No - error, no row, a permanent hole in a paid dataset. Pruned, so the next run - re-fetches it. - - The second case is the dangerous one: the first costs money re-downloading data you - already have, the second leaves you believing you have data you do not. - - **Empty-window advance markers are not ghosts and are never pruned.** A paged pull - that meets an empty window writes ``{last_date: ...}`` with no parquet on purpose, - so a re-run does not refetch that span forever; for a symbol whose data starts after - the GLBX floor, the marker is all there is. Pruning on "no file" alone would delete - them and restore the loop, so the test is "claims rows AND has no file". See - ``_claims_rows``. - - Reads only local files, never the API. Returns - ``{"recorded": {key: last_date}, "pruned": [key, ...]}``. - """ - manifest = _load_ingest_manifest() - recorded: dict = {} - for schema in _SCHEMAS: - sub = "ohlcv" if schema == "ohlcv-1d" else "statistics" - d = raw_root() / sub - if not d.exists(): - continue - for p in sorted(d.glob("*.parquet")): - name = p.stem # "", e.g. "6E.n.0" - feed = next((f for f in _FEEDS if name.endswith(f)), None) - if feed is None: - continue - symbol = name[: -len(feed)] - try: - df = pd.read_parquet(p) - except Exception as e: # noqa: BLE001 - print(f"reconcile: could not read {p.name} — {e}") - continue - if df.empty: - continue - first, newest = _date_bounds(df, schema) - key = f"{symbol}{feed}:{schema}" - manifest[key] = { - "first_date": first, - "last_date": newest, - "n_rows": int(len(df)), - "fetched_at": dt.datetime.utcnow().isoformat(timespec="seconds") + "Z", - "reconciled": True, - } - recorded[key] = newest - - pruned: list = [] - if prune: - for key in sorted(manifest): - try: - sym_feed, schema = key.rsplit(":", 1) - except ValueError: - continue - sub = "ohlcv" if schema == "ohlcv-1d" else "statistics" - if (raw_root() / sub / f"{sym_feed}.parquet").exists(): - continue - if not _claims_rows(manifest[key]): - # NOT a ghost. A paged pull that meets an empty window writes a - # `last_date` advance marker with no parquet, deliberately, so a - # re-run does not refetch that empty span forever (see the - # `elif chunk_days` branch in _ingest_range). For a symbol whose - # data starts after the GLBX floor, every early window is empty - # and the marker is ALL there is. - # - # Pruning on "no file" alone would delete exactly those markers and - # restore the refetch loop they exist to prevent, on every run, - # against a paid API. This function exists to stop a silent hole in - # paid data; deleting a marker would trade it for a silent spend. - continue - pruned.append(key) - for key in pruned: - del manifest[key] - - _write_ingest_manifest(manifest) - return {"recorded": recorded, "pruned": pruned} - - -# ── Batch ingest: robust large pulls via the databento batch API ───────────── -# The streaming timeseries.get_range chokes on from-inception statistics (read timeouts, -# dropped streams). The batch API prepares result files server-side and delivers them as -# a download (robust, resumable), which is what it is designed for. Same raw store + -# manifest as the streaming ingest, so build()/reconcile() are unchanged. Statistics are -# fetched FULL here (no windowing — batch handles the volume). - -def _batch_fetch(client, dataset, dbsyms, schema, start, end, *, poll=30, timeout=14400) -> dict: - """Submit one databento BATCH job for many continuous symbols over [start, end), wait - for it, download the CSV(s), and return ``{dbsym: raw_df}`` grouped by the ``symbol`` - column. Raises on job failure/timeout.""" - import tempfile - - def _jid(j): - return j.get("id") if isinstance(j, dict) else getattr(j, "id", None) - - def _state(j): - return (j.get("state") if isinstance(j, dict) else getattr(j, "state", None)) or "unknown" - - job = client.batch.submit_job( - dataset=dataset, symbols=list(dbsyms), stype_in="continuous", schema=schema, - start=start, end=end, encoding="csv", split_symbols=False, delivery="download") - job_id = _jid(job) - if not job_id: - raise RuntimeError("batch submit_job returned no job id") - - waited = 0 - while True: - me = next((j for j in client.batch.list_jobs() if _jid(j) == job_id), None) - st = _state(me) - if st == "done": - break - if st == "expired" or "fail" in str(st).lower(): - raise RuntimeError(f"batch job {job_id} state={st}") - if waited >= timeout: - raise TimeoutError(f"batch job {job_id} not done after {timeout}s (state={st})") - time.sleep(poll) - waited += poll - - frames: dict = {} - with tempfile.TemporaryDirectory() as tmp: - for path in client.batch.download(job_id=job_id, output_dir=tmp): - if ".csv" not in str(path): - continue - try: - df = pd.read_csv(path) # pandas infers .zst/.gz compression - except Exception as e: # noqa: BLE001 - print(f"batch: could not read {path} — {e}") - continue - if df.empty or "symbol" not in df.columns: - continue - for dbsym, g in df.groupby("symbol"): - frames.setdefault(dbsym, []).append(g) - return {k: pd.concat(v, ignore_index=True) for k, v in frames.items()} - - -def _normalize_batch_csv(df: pd.DataFrame, schema: str) -> pd.DataFrame: - """Bring a batch CSV frame to the same shape the streaming ``_normalize`` produces - (ohlcv: tz-naive daily Date index; statistics: tz-naive columns).""" - df = df.copy() - for c in ("ts_event", "ts_ref"): - if c in df.columns: - df[c] = _to_naive(df[c]) - if schema == "ohlcv-1d": - df = df.set_index("ts_event") - df.index = df.index.normalize() - df.index.name = "Date" - return df[~df.index.duplicated(keep="last")].sort_index() - return df.drop_duplicates().reset_index(drop=True) - - -def ingest_batch(symbols=None, *, client=None, dataset="GLBX.MDP3", end=None, - cold_start=GLBX_HISTORY_FLOOR, poll=30) -> dict: - """Batch-API variant of ingest(): fetch each (feed, schema) as a databento batch job - (download-to-file) instead of streaming — robust for large from-inception pulls. One - job per (feed, schema) covers every symbol still needing it, from the earliest resume - point; per-symbol rows are appended and the manifest advanced individually, so resume - and build are unchanged. Returns {kind, ok, symbols, rows}.""" - targets = [s for s in all_symbols() - if s.databento and s.price_source in (None, "databento") - and (symbols is None or s.internal in symbols)] - if not targets: - print("databento batch ingest: no databento-capable symbols" - + (f" among {symbols}" if symbols else "")) - return {"kind": "ingest_databento", "ok": True, "symbols": 0, "rows": 0} - if client is None: - client = _client_from_env() - - end = end or (pd.Timestamp.now().normalize() - pd.Timedelta(days=1)).strftime("%Y-%m-%d") - end_ts = pd.Timestamp(end) - manifest = _load_ingest_manifest() - total_rows, failed = 0, 0 - run_start = time.monotonic() - - for feed in _FEEDS: - for schema in _SCHEMAS: - needed, starts = {}, [] - for s in targets: - last = manifest.get(f"{s.internal}{feed}:{schema}", {}).get("last_date") - st = (pd.Timestamp(last) + pd.Timedelta(days=1)) if last else pd.Timestamp(cold_start) - if st < pd.Timestamp(GLBX_HISTORY_FLOOR): - st = pd.Timestamp(GLBX_HISTORY_FLOOR) - if st >= end_ts: - continue # already current - needed[f"{s.databento}{feed}"] = s.internal - starts.append(st) - if not needed: - continue - job_start = min(starts).strftime("%Y-%m-%d") - t0 = time.monotonic() - print(f"batch {feed} {schema}: {len(needed)} symbols {job_start}..{end} — submitting job...") - try: - frames = _batch_fetch(client, dataset, list(needed), schema, job_start, end, poll=poll) - except Exception as e: # noqa: BLE001 — batch/network is flaky - print(f"batch {feed} {schema}: FAILED — {e}") - failed += 1 - continue - wrote = 0 - for dbsym, raw in frames.items(): - internal = needed.get(dbsym) - if internal is None: - continue - raw = _normalize_batch_csv(raw, schema) - if raw.empty: - continue - _append_raw(internal, feed, schema, raw) - first, newest = _date_bounds(raw, schema) - key = f"{internal}{feed}:{schema}" - rec = manifest.get(key, {}) - manifest[key] = { - "first_date": rec.get("first_date") or first, - "last_date": newest, - "n_rows": int(rec.get("n_rows", 0)) + len(raw), - "fetched_at": dt.datetime.utcnow().isoformat(timespec="seconds") + "Z", - "batch": True, - } - wrote += len(raw) - _write_ingest_manifest(manifest) - total_rows += wrote - print(f"batch {feed} {schema}: +{wrote:,} rows for {len(frames)} symbols in " - f"{_fmt_hms(time.monotonic() - t0)}") - - print(f"databento batch ingest: {len(targets)} symbols, {total_rows:,} rows in " - f"{_fmt_hms(time.monotonic() - run_start)}" - + (f" ({failed} job failure(s))" if failed else "")) - return {"kind": "ingest_databento", "ok": failed == 0, "symbols": len(targets), "rows": total_rows} - - -# ── Build (Stage 2): free, raw store -> back-adjusted store prices ──────────── -# ADR-0006. Derives two series per symbol from the local raw store (no API cost): -# unadj — raw front continuous (.n.0), settlement close, roll gaps intact. -# backadj — additive back-adjustment, Norgate's method exactly: at each roll the -# gap = new_close - old_close on the roll date = n1_settle - n0_settle; -# every price up to AND INCLUDING the roll date is shifted by the gap; -# gaps accumulate back-to-front so the newest segment stays at real prices. -# Then store.write_prices(..., source='databento'). propadj is derived on read from -# unadj + backadj by the consumer API, unchanged. - -_OHLCV_COLMAP = {"open": "Open", "high": "High", "low": "Low", - "close": "Close", "volume": "Volume"} -_OUT_COLS = ["Open", "High", "Low", "Close", "Volume", "Open Interest"] - -# Databento reports settlement in true dollars, but the toolchain's price convention -# (inherited from Norgate, which the rest of the stack was built on) quotes a handful of -# contracts in cents / the IMM x100 form: silver and copper in cents/unit, JPY in the IMM -# quote. Scale those at build time so the databento store is a drop-in for the Norgate one. -# Applied to the price columns only (never Volume/Open Interest, which are counts). The raw -# bronze store stays faithful to databento; this is a silver-stage reconciliation, so it -# costs nothing to change. Verified by scripts/validate_databento_vs_norgate.py: with the -# scale on, SI/HG/6J daily-change correlation is >0.998 and scale_ratio ~1.0 vs Norgate. -_PRICE_SCALE = {"SI": 100.0, "HG": 100.0, "6J": 100.0} - - -def _read_ohlcv(symbol: str, feed: str) -> pd.DataFrame: - p = _raw_path(symbol, feed, "ohlcv-1d") - if not p.exists(): - return pd.DataFrame() - df = pd.read_parquet(p).rename(columns=_OHLCV_COLMAP) - # instrument_id is the roll signal: for a continuous series databento's `symbol` - # column is the constant alias ("ES.n.0"), while instrument_id changes to the new - # contract at each roll. Keep both. - keep = [c for c in ("Open", "High", "Low", "Close", "Volume", "instrument_id", "symbol") - if c in df.columns] - df = df[keep].copy() - df.index = pd.to_datetime(df.index).normalize() - df.index.name = "Date" - return df[~df.index.duplicated(keep="last")].sort_index() - - -def _stat_series(symbol: str, feed: str, stat_type: int, date_col: str, value_col: str) -> pd.Series: - """Daily series of a databento statistic — settlement (stat_type 3, dated by - ts_ref, the session it applies to) or Open Interest (stat_type 9, by ts_event).""" - p = _raw_path(symbol, feed, "statistics") - if not p.exists(): - return pd.Series(dtype="float64") - st = pd.read_parquet(p) - if not {"stat_type", date_col, value_col}.issubset(st.columns): - return pd.Series(dtype="float64") - sel = st[st["stat_type"] == stat_type].copy() - if sel.empty: - return pd.Series(dtype="float64") - sel["D"] = pd.to_datetime(sel[date_col]).dt.normalize() - return sel.groupby("D")[value_col].last() - - -def _with_settlement(ohlcv: pd.DataFrame, settle: pd.Series) -> pd.DataFrame: - """Override Close with exchange settlement (stat_type 3) where present, so the - series is settlement-based like Norgate rather than the ohlcv last-trade close.""" - if ohlcv.empty or settle.empty: - return ohlcv - out = ohlcv.copy() - out["Close"] = settle.reindex(out.index).combine_first(out["Close"]) - return out - - -def _roll_key(df: pd.DataFrame) -> Optional[str]: - """The column that identifies the active contract, for roll detection. Prefer - ``instrument_id`` (changes at each roll); ``symbol`` is only useful when it carries - resolved contracts rather than databento's constant continuous alias.""" - return "instrument_id" if "instrument_id" in df.columns else ( - "symbol" if "symbol" in df.columns else None) - - -def _cumulative_offset(n0: pd.DataFrame, n1: pd.DataFrame): - """Additive back-adjust offset per date: the sum of the roll gap - (n1_close - n0_close measured ON each roll date) over all rolls at or after that - date. A roll date is the last session a front contract is active — its active - contract (instrument_id) differs from the next day's. Returns (offset Series on - n0.index, n_rolls, n_missing).""" - offset = pd.Series(0.0, index=n0.index) - key = _roll_key(n0) - if key is None or n1.empty or "Close" not in n1.columns: - return offset, 0, 0 - sym = n0[key] - is_roll = sym.ne(sym.shift(-1)) & sym.shift(-1).notna() - roll_dates = list(n0.index[is_roll]) - if not roll_dates: - return offset, 0, 0 - n0_close, n1_close = n0["Close"], n1["Close"].reindex(n0.index) - gaps, missing = {}, 0 - for d in roll_dates: - g = n1_close.get(d, float("nan")) - n0_close.get(d, float("nan")) - if pd.isna(g): - missing += 1 - continue - gaps[d] = float(g) - # Walk dates newest->oldest; add each roll's gap as we pass it (inclusive of the - # roll date), so every earlier price carries the cumulative offset. - running, out = 0.0, {} - for d in reversed(list(n0.index)): - if d in gaps: - running += gaps[d] - out[d] = running - return pd.Series(out).reindex(n0.index), len(gaps), missing - - -def build(symbols=None) -> dict: - """Stage 2 (free): read the raw store and write unadj + backadj daily bars to the - cotdata store for every databento-capable symbol. Requires the raw store populated - by ingest(). Returns {kind, ok, symbols, wrote}.""" - targets = [s for s in all_symbols() - if s.databento and s.price_source in (None, "databento") - and (symbols is None or s.internal in symbols)] - wrote, skipped = 0, 0 - for s in targets: - n0 = _read_ohlcv(s.internal, ".n.0") - if n0.empty: - print(f"{s.internal}: no raw .n.0 ohlcv — run ingest first; skipping") - skipped += 1 - continue - n1 = _read_ohlcv(s.internal, ".n.1") - n0 = _with_settlement(n0, _stat_series(s.internal, ".n.0", 3, "ts_ref", "price")) - if not n1.empty: - n1 = _with_settlement(n1, _stat_series(s.internal, ".n.1", 3, "ts_ref", "price")) - # Reconcile databento's dollar units to the toolchain's Norgate convention BEFORE - # the roll-gap math, so unadj, the gaps, and backadj all end up in the same units. - scale = _PRICE_SCALE.get(s.internal) - if scale: - for df in (n0, n1): - for c in ("Open", "High", "Low", "Close"): - if c in df.columns: - df[c] = df[c] * scale - oi = _stat_series(s.internal, ".n.0", 9, "ts_event", "quantity") - - unadj = n0.copy() - unadj["Open Interest"] = oi.reindex(unadj.index) if not oi.empty else float("nan") - extra = [] - key = _roll_key(unadj) - if key: - # The active-contract id, as a string, so roll_dates() can detect the change. - # (databento gives no calendar month here, so this is the instrument_id.) - unadj["Delivery Month"] = unadj[key].astype(str) - extra = ["Delivery Month"] - - offset, n_rolls, n_missing = _cumulative_offset(n0, n1) - if n_rolls == 0: - print(f"{s.internal}: no rolls detected (backadj == unadj) — verify the raw ohlcv " - f"carries `instrument_id` (the roll signal; `symbol` is the constant alias)") - if n_missing: - print(f"{s.internal}: {n_missing} roll gap(s) unmeasurable (no .n.1 close) — treated as 0") - - backadj = unadj.copy() - for c in ("Open", "High", "Low", "Close"): - if c in backadj.columns: - backadj[c] = backadj[c] + offset - - for c in _OUT_COLS: - if c not in unadj.columns: - unadj[c] = float("nan") - backadj[c] = float("nan") - cols = _OUT_COLS + extra - store.write_prices(s.internal, "unadj", unadj[cols], source="databento") - store.write_prices(s.internal, "backadj", backadj[cols], source="databento") - wrote += 1 - print(f"{s.internal}: built unadj+backadj ({len(unadj)} bars, {n_rolls} rolls) -> store") - - return {"kind": "build_databento", "ok": skipped == 0, "symbols": len(targets), "wrote": wrote} diff --git a/src/cotdata/registry.py b/src/cotdata/registry.py index 4769afd..cf315c2 100644 --- a/src/cotdata/registry.py +++ b/src/cotdata/registry.py @@ -1,17 +1,23 @@ -"""The single symbol registry: internal ↔ Norgate ↔ CFTC code ↔ asset class. -Replaces the scattered maps (CotSymbolCodeMap, the databento_mapping). +"""The single symbol registry: internal symbol ↔ CFTC code ↔ asset class. +Replaces the scattered maps (CotSymbolCodeMap and friends). Scope = FIXED IDENTITY facts only (never change). TUNABLE strategy parameters (positioning-index CustomLookbackWeeks, thresholds, TV chart symbols) stay in cot-analyzer/config/params.yaml — the data layer must not carry strategy knobs. +CFTC POSITIONING ONLY, since ADR-0007. The price-vendor mappings this file used to +carry (norgate, yahoo, databento) live in marketdata's registry now, beside the +producers that read them. The two registries are separate files and cannot be +merged: this loader requires a `cftc_code` of every symbol, and equities have no +COT report — which is the same requirement that made a separate package necessary +in the first place. + The table lives in registry.yaml (asset_class -> symbol -> attrs), loaded at import. Point $COTDATA_REGISTRY at an alternate file to override it. Sources: • cftc_code — cot-analyzer CotSymbolCodeMap • asset_class — CotIndexer asset classes - • norgate — '&' + CME root (e.g., "&ES"); required by norgatedata.price_timeseries(). is_equity is True only for the four equity indices (a fixed classification the equity-vs-commodity rules key off); it is derived from asset_class == "Equities" unless a symbol overrides it explicitly. @@ -27,27 +33,18 @@ @dataclass(frozen=True) class Symbol: internal: str # pipeline root, e.g. "ES" - norgate: Optional[str] # Norgate continuous symbol, e.g. "&ES"; None when - # Norgate has no series (norgate: null in YAML) — - # the Norgate producer skips these (priced elsewhere) asset_class: str is_equity: bool report_type: str = "disagg" # "tff" for financials, "disagg" for commodities cftc_code: Optional[str] = None - # Optional Yahoo Finance ticker for markets Norgate/databento don't cover (the - # yfinance provider prices these; e.g. "EEM"/"EFA" as ETF proxies for MSCI EM/EAFE). - yahoo: Optional[str] = None - # Databento GLBX.MDP3 continuous root — the databento producer queries - # ".n.0". Defaults to the internal symbol; None (registry - # `databento: null`) when GLBX carries no series for the market (ICE softs, - # lumber, MSCI intl). None is the capability signal the databento producer - # filters on, and the deployment falls back to yfinance where a yahoo ticker exists. - databento: Optional[str] = None - # Optional per-symbol price-source override (norgate | databento | yfinance). - # Normally unset: the source is the deployment default (COTDATA_PRICE_SOURCE) - # resolved against capability (see resolve_source). Set only to pin one market - # to a specific vendor regardless of the deployment default. - price_source: Optional[str] = None + # NO VENDOR MAPPINGS, and none should come back. The norgate / yahoo / databento + # columns, the `price_source` override and the whole resolve_source machinery moved + # to marketdata's registry with the producers that read them (ADR-0007). + # + # An earlier note here claimed they had to stay because a deployment might SHARE one + # registry file between the two packages via COTDATA_REGISTRY. That was wrong: this + # loader hard-requires `cftc_code` and marketdata's equities do not have one, so the + # two files can never be the same file. # Predecessor CFTC codes from earlier exchange/contract listings of the SAME # instrument, stitched in chronologically behind cftc_code by get_cot. Each # entry is a bare code string (scale 1.0) or a (code, scale) tuple. Kept as a @@ -69,23 +66,6 @@ def _coerce_hist_codes(raw) -> tuple: return tuple(tuple(h) if isinstance(h, list) else h for h in (raw or [])) -# The price vendors a symbol can be sourced from. 'yfinance' is the universal -# research-grade fallback when the deployment's preferred vendor can't serve a market. -PRICE_SOURCES = ("norgate", "databento", "yfinance") - - -def _validate_source(value, internal) -> Optional[str]: - """Validate an explicit price_source override from the YAML (None if unset).""" - if value is None: - return None - v = str(value).strip().lower() - if v not in PRICE_SOURCES: - raise ValueError( - f"cotdata registry: symbol '{internal}' has price_source={value!r}; " - f"expected one of {PRICE_SOURCES}.") - return v - - def load_registry(yaml_path=None) -> Dict[str, Symbol]: """Build the symbol registry from YAML. @@ -132,7 +112,6 @@ def load_registry(yaml_path=None) -> Dict[str, Symbol]: f"cotdata registry: symbol '{internal}' is missing cftc_code.") registry[internal] = Symbol( internal=internal, - norgate=attrs.get("norgate", f"&{internal}"), asset_class=asset_class, # Derived from the class so the two can't drift; an explicit # is_equity in the YAML still wins if a symbol ever needs it. @@ -141,11 +120,6 @@ def load_registry(yaml_path=None) -> Dict[str, Symbol]: report_type=attrs.get("report_type", "tff" if asset_class in ("Equities", "FX", "Rates") else "disagg"), cftc_code=attrs["cftc_code"], hist_codes=_coerce_hist_codes(attrs.get("hist_codes")), - yahoo=attrs.get("yahoo"), - # Defaults to the internal root; explicit `databento: null` marks a - # market GLBX doesn't carry (capability signal for the producer). - databento=attrs.get("databento", internal), - price_source=_validate_source(attrs.get("price_source"), internal), ) return registry @@ -164,55 +138,3 @@ def all_symbols() -> List[Symbol]: def by_asset_class(asset_class: str) -> List[Symbol]: return [s for s in REGISTRY.values() if s.asset_class == asset_class] - - -# ── Price-source selection ─────────────────────────────────────────────────── -# Which vendor prices a symbol is a DEPLOYMENT choice, not a fixed identity fact: -# the same ES is Norgate for local research and databento on the public-dash server. -# So it is resolved at runtime from three inputs, not baked per-symbol in the shared -# registry: (1) a deployment default (COTDATA_PRICE_SOURCE), (2) per-symbol capability -# (which vendors carry a series — the norgate/databento/yahoo mappings), and (3) an -# optional per-symbol override. See ADR-0006. -# -# ADR-0007 NOTE. Only the `databento` mapping has a producer left in this package — -# `norgate` and `yahoo` moved to `marketdata`, which resolves them against its OWN -# registry. They stay here because this file is a CAPABILITY map that a deployment may -# share between the two packages via COTDATA_REGISTRY, and because deleting a column -# from a shared registry breaks the other reader. So `resolve_source` can still name a -# vendor this package cannot produce: that is a statement about the market, not a -# promise that `cotdata-update` will fetch it. - -def _can_serve(sym: Symbol, source: str) -> bool: - """Whether `source` has a price series for `sym` (its vendor mapping is present).""" - if source == "norgate": - return sym.norgate is not None - if source == "databento": - return sym.databento is not None - if source == "yfinance": - return sym.yahoo is not None - raise ValueError(f"unknown price source {source!r}; expected one of {PRICE_SOURCES}") - - -def resolve_source(sym: Symbol, default: str = "norgate") -> Optional[str]: - """The vendor that prices `sym` on a deployment whose default source is `default`. - - An explicit `sym.price_source` override wins. Otherwise use `default` when that - vendor can serve the symbol, else fall back to yfinance where a yahoo ticker - exists. Returns None when nothing can price it (the caller skips the symbol).""" - if sym.price_source: - return sym.price_source - if _can_serve(sym, default): - return default - if _can_serve(sym, "yfinance"): - return "yfinance" - return None - - -def default_price_source() -> str: - """Deployment-wide default price vendor from $COTDATA_PRICE_SOURCE ('norgate' if - unset). Local research leaves it unset; the databento server sets it to 'databento'.""" - src = os.environ.get("COTDATA_PRICE_SOURCE", "norgate").strip().lower() - if src not in PRICE_SOURCES: - raise ValueError( - f"COTDATA_PRICE_SOURCE={src!r} is invalid; expected one of {PRICE_SOURCES}.") - return src diff --git a/src/cotdata/registry.yaml b/src/cotdata/registry.yaml index 038154a..92a198a 100644 --- a/src/cotdata/registry.yaml +++ b/src/cotdata/registry.yaml @@ -15,19 +15,13 @@ Equities: cftc_code: "33874A" NKD: # Nikkei 225 ($-denominated, CME) cftc_code: "240741" - # MSCI international indices — the liquid out-of-selection held-out markets. Priced - # off their ETF proxies via Yahoo (Norgate/databento don't cover them; ETFs have no - # roll gaps). COT is the real futures positioning (ICE Futures U.S.). - MME: # MSCI Emerging Markets Index (ICE) — priced via EEM + # MSCI international indices — the liquid out-of-selection held-out markets. COT is + # the real futures positioning (ICE Futures U.S.). They have no futures bars anywhere + # in the fleet: marketdata prices them off ETF proxies and its registry omits them. + MME: # MSCI Emerging Markets Index (ICE) cftc_code: "244042" - norgate: null # no Norgate continuous series — priced off the ETF proxy - databento: null # ICE product — not on GLBX.MDP3 - yahoo: "EEM" - MFS: # MSCI EAFE Index (ICE) — priced via EFA + MFS: # MSCI EAFE Index (ICE) cftc_code: "244041" - norgate: null # (Norgate doesn't carry MSCI intl indices) — skip in the - databento: null # ICE product — not on GLBX.MDP3 - yahoo: "EFA" # Norgate producer; the yfinance provider prices these Metals: GC: @@ -56,7 +50,6 @@ Energies: # docs/design/amendments-2026-08-03.md §C15). Norgate carries it as &WBS. WBS: # WTI Crude Oil (ICE Futures Europe) cftc_code: "067411" - databento: null # ICE product — not on GLBX.MDP3 Grains: ZC: @@ -107,8 +100,6 @@ Currencies: cftc_code: "112741" DX: cftc_code: "098662" - databento: null # ICE product (US Dollar Index), not on GLBX.MDP3 - yahoo: "DX-Y.NYB" # cash DXY (ICE). Yahoo has no working DX futures ticker Fixed Income: ZN: @@ -120,35 +111,19 @@ Fixed Income: ZB: cftc_code: "020601" -# Softs trade on ICE, not CME Globex, so databento GLBX.MDP3 carries no series for -# them (see providers/databento.py _DATABENTO_UNSUPPORTED). Norgate covers them for -# local research; a databento deployment falls back to the Yahoo continuous futures -# below (research-grade). LBR's Yahoo symbol is unverified, confirm before relying on it. Softs: SB: cftc_code: "080732" - databento: null - yahoo: "SB=F" CT: cftc_code: "033661" - databento: null - yahoo: "CT=F" CC: cftc_code: "073732" - databento: null - yahoo: "CC=F" KC: cftc_code: "083731" - databento: null - yahoo: "KC=F" OJ: cftc_code: "040701" - databento: null - yahoo: "OJ=F" LBR: - cftc_code: "058644" - databento: null - yahoo: "LBR=F" # CME Lumber (physical), listed 2022-08. LBS=F was the delisted + cftc_code: "058644" # CME Lumber (physical), listed 2022-08 # random-length contract (dark since 2023-05), verified stale. hist_codes: - ["058643", 4.0] diff --git a/src/cotdata/store.py b/src/cotdata/store.py index da2d555..959468b 100644 --- a/src/cotdata/store.py +++ b/src/cotdata/store.py @@ -28,25 +28,6 @@ def _atomic_write_parquet(df: pd.DataFrame, path: Path) -> None: os.remove(tmp) -# ── Prices ──────────────────────────────────────────────────────────────── -# ADR-0007 makes this package COT-only, and the CONSUMER bar API (get_prices, -# roll_dates, the derived propadj tier) is gone — read bars from `marketdata`. -# -# These two survive because the databento provider does not have a marketdata -# equivalent yet and still writes here (ADR-0006: an independently-built -# alternative to Norgate, and the source for the intraday news-failure work). -# Keeping the write path without the read path would make its output unreadable -# through the library, so the store-level pair stays; only the consumer API left. -def write_prices(symbol: str, adjustment: str, df: pd.DataFrame, source: str) -> None: - _atomic_write_parquet(df, config.prices_dir() / f"{symbol}_{adjustment}.parquet") - _touch_manifest("prices", f"{symbol}_{adjustment}", df, source) - - -def read_prices(symbol: str, adjustment: str) -> pd.DataFrame: - p = config.prices_dir() / f"{symbol}_{adjustment}.parquet" - return pd.read_parquet(p) if p.exists() else pd.DataFrame() - - # ── COT Legacy ──────────────────────────────────────────────────────────── def write_cot_legacy(name: str, df: pd.DataFrame, source: str) -> None: _atomic_write_parquet(df, config.cot_legacy_dir() / f"{name}.parquet") @@ -102,11 +83,13 @@ def read_cot_supplemental(name: str) -> pd.DataFrame: # half, and they never touch the same manifest file. Listing every domain here (and # refusing unknown ones) is what stops a new domain quietly joining the wrong side. # -# `metadata` (futures contract specs) is still DECLARED but nothing writes it any -# more: ADR-0007 moved contract specs to marketdata along with the bars. It stays -# on the map because stores written before that move carry `metadata` entries, and -# an undeclared domain is skipped by migrate_manifests() — which would strand those -# entries in the legacy aggregate forever. Read-only legacy; do not add writers. +# `prices` and `metadata` are still DECLARED and nothing writes either any more: +# ADR-0007 moved every bar and the contract-specs table to marketdata, and the +# databento producer followed them. They stay on the map because stores written +# before those moves carry such entries, and an undeclared domain is SKIPPED by +# migrate_manifests() — which would strand them in the legacy aggregate forever, +# and make reconcile_manifest() resolve their directory by fallback rather than by +# declaration. Read-only legacy on both: do not add a writer to either. HALVES = ("cot", "prices") _DOMAIN_HALF = { "prices": "prices", @@ -132,10 +115,10 @@ def half_for(kind: str) -> str: def _empty_manifest() -> dict: - # No `metadata` key: a store created from here on never gets contract specs - # (they live in marketdata now). Legacy stores that have them still read fine — + # COT domains only: a store created from here on never gets bars or contract + # specs (both live in marketdata). Legacy stores that have them still read fine — # load_manifest() overlays whatever domains it finds on disk. - return {"schema_version": config.SCHEMA_VERSION, "prices": {}, + return {"schema_version": config.SCHEMA_VERSION, "cot_legacy": {}, "cot_disagg": {}, "cot_tff": {}, "cot_supplemental": {}} diff --git a/src/cotdata/update.py b/src/cotdata/update.py index 2c682e5..242ee3e 100644 --- a/src/cotdata/update.py +++ b/src/cotdata/update.py @@ -1,74 +1,20 @@ """Producer CLI: cotdata-update --cot-all - cotdata-update --build-databento --symbols ES NQ cotdata-update --check # read-only store status cotdata-update --reconcile # prune stale manifest ghosts Writes to $COTDATA_STORE. Schedule COT weekly (Friday, after the CFTC release). -ADR-0007 moved bar production out of this package: the Norgate and Yahoo price -producers, and the consumer bar API, now live in `marketdata` — run -`marketdata-update --bars` on the nightly job. What remains on the price side is -databento, which has no marketdata equivalent yet (ADR-0006: an independently -built alternative to Norgate, and the source for the intraday work).""" +ADR-0007 is complete: every price producer left this package. Norgate, Yahoo and +databento all live in `marketdata` now, against `$MARKETDATA_STORE` — run +`marketdata-update --bars` (or `--ingest-databento` / `--build-databento`) there. +This CLI fetches CFTC positioning and nothing else.""" import argparse import datetime as _dt from . import config -# Which producer half each action belongs to. `cotdata-cot` and `cotdata-prices` refuse -# the other half's actions, so a host can be given exactly one job. That matters most on -# a second producer machine: a Windows price box that could also run --cot-all would -# become a second COT producer racing the first, which is the failure the split manifests -# exist to contain. Read-only actions (--check, --reconcile) belong to both. -_HALF_ACTIONS = { - "cot": ("cot_legacy", "cot_disagg", "cot_tff", "cot_supplemental", "cot_all"), - "prices": ("ingest_databento", "build_databento"), -} - - -def _reject_other_half(parser, args, half: str) -> None: - other = "prices" if half == "cot" else "cot" - used = [a for a in _HALF_ACTIONS[other] if getattr(args, a, False)] - if used: - flags = ", ".join("--" + a.replace("_", "-") for a in used) - parser.error( - f"{flags} belong(s) to the {other} half. This entry point runs the {half} " - f"half only, so one host does one job. Use cotdata-{other} for those, or " - f"cotdata-update to run both from one machine.") - - -# Parser dests that are NOT producer actions: read-only or one-shot maintenance, and -# modifiers on another flag. Everything else must be classified into a half above — -# test_every_action_flag_is_assigned_to_a_half reads both lists to enforce it, so a new -# action flag fails the suite until someone decides which side of the seam it is on. -_NON_ACTIONS = frozenset({ - "help", "symbols", "check", "migrate_manifests", "reconcile", - "reconcile_databento", "windowed_n1_stats", "batch", -}) - def _parser() -> argparse.ArgumentParser: p = argparse.ArgumentParser(description="cotdata producer — fetch sources into the store.") - p.add_argument("--ingest-databento", action="store_true", - help="Databento Stage 1 (paid API, cross-platform): fetch raw .n.0/.n.1 " - "ohlcv-1d + statistics into the append-only raw store ($COTDATA_DATABENTO_RAW). " - "The statistics schema is paged in yearly windows so a from-inception " - "pull can't time out in one request. Resumable — re-runs (and a mid-pull " - "failure) only pull the missing dates. Needs DATABENTO_API_KEY.") - p.add_argument("--build-databento", action="store_true", - help="Databento Stage 2 (free, no API): build back-adjusted prices from the " - "raw store into the cotdata store. Run after --ingest-databento.") - p.add_argument("--windowed-n1-stats", nargs="?", type=int, const=3, default=None, metavar="DAYS", - help="With --ingest-databento: fetch the n.1 statistics schema only in " - "±DAYS windows around roll dates (default 3) instead of full history. " - "n.1 settlement is only needed at rolls, so this cuts the biggest " - "avoidable download with no accuracy loss. Recommended for the backfill.") - p.add_argument("--batch", action="store_true", - help="With --ingest-databento: use the databento BATCH API (prepare + " - "download files) instead of the (default) paged streaming ingest. " - "NOTE: submitting a single from-inception continuous job can 504 at " - "databento's gateway (heavy server-side cost/resolution over 16y), so " - "the paged streaming default is preferred for cold-start backfills; " - "reach for --batch only for a bounded catch-up.") p.add_argument("--cot-legacy", action="store_true", help="Update CFTC COT Legacy (cross-platform).") p.add_argument("--cot-disagg", action="store_true", help="Update CFTC COT Disaggregated Futures-Only (cross-platform).") p.add_argument("--cot-tff", action="store_true", help="Update Traders in Financial Futures (TFF) COT (cross-platform).") @@ -89,21 +35,12 @@ def _parser() -> argparse.ArgumentParser: p.add_argument("--reconcile", action="store_true", help="Prune manifest entries whose parquet file is missing (ghosts " "from old naming), refresh status.json, and exit. Never touches data.") - p.add_argument("--reconcile-databento", action="store_true", - help="Make the databento ingest manifest match the parquet files on disk, " - "in BOTH directions: record tables an interrupted run left " - "unrecorded (so a restart does not re-download them), and prune " - "entries whose file is missing (so a restart does not skip them as " - "'already current' and leave a silent hole). Local files only, no " - "API. Exit.") return p -def main(argv=None, half=None) -> None: +def main(argv=None) -> None: p = _parser() args = p.parse_args(argv) - if half: - _reject_other_half(p, args, half) config.store_root() # fail fast if COTDATA_STORE unset @@ -143,52 +80,15 @@ def main(argv=None, half=None) -> None: "at": _dt.datetime.utcnow().isoformat(timespec="seconds") + "Z"}) return - if args.reconcile_databento: - from .providers import databento - res = databento.reconcile_manifest() - recorded, pruned = res["recorded"], res["pruned"] - if not recorded and not pruned: - print("databento reconcile: manifest already matches the raw store.") - if recorded: - syms = sorted({k.split(".n.")[0] for k in recorded}) - print(f"databento reconcile: recorded {len(recorded)} raw table(s) across " - f"{len(syms)} symbol(s) into the ingest manifest.") - print(" symbols: " + ", ".join(syms)) - if pruned: - syms = sorted({k.split(".n.")[0] for k in pruned}) - print(f"databento reconcile: PRUNED {len(pruned)} manifest entr" - f"{'y' if len(pruned) == 1 else 'ies'} with no parquet on disk, across " - f"{len(syms)} symbol(s). The next --ingest-databento will re-fetch these; " - f"without the prune it would have skipped them as 'already current'.") - print(" symbols: " + ", ".join(syms)) - return - - if not (args.ingest_databento or args.build_databento or args.cot_legacy - or args.cot_disagg or args.cot_tff or args.cot_supplemental or args.cot_all): - p.error("nothing to do — pass --check, --ingest-databento, --build-databento, " - "--cot-legacy, --cot-disagg, --cot-tff, --cot-supplemental, or " - "--cot-all. Norgate/Yahoo bars moved to marketdata (ADR-0007): use " - "'marketdata-update --bars'.") + if not (args.cot_legacy or args.cot_disagg or args.cot_tff + or args.cot_supplemental or args.cot_all): + p.error("nothing to do — pass --check, --cot-legacy, --cot-disagg, --cot-tff, " + "--cot-supplemental or --cot-all. Every price producer moved to " + "marketdata (ADR-0007): use 'marketdata-update --bars', or " + "'--ingest-databento'/'--build-databento' there.") kinds = [] failed_kinds = [] # domains that hard-failed → non-zero exit so a scheduler retries - if args.ingest_databento: - from .providers import databento - if args.batch: - r = databento.ingest_batch(symbols=args.symbols) - else: - r = databento.ingest(symbols=args.symbols, n1_stats_window=args.windowed_n1_stats) - kinds.append("ingest_databento") - if not (r or {}).get("ok", True): - failed_kinds.append("ingest_databento") - - if args.build_databento: - from .providers import databento - r = databento.build(symbols=args.symbols) - kinds.append("build_databento") - if not (r or {}).get("ok", True): - failed_kinds.append("build_databento") - if args.cot_legacy or args.cot_all: from .providers import cftc r = cftc.update() @@ -228,10 +128,9 @@ def main(argv=None, half=None) -> None: # Structured heartbeat for downstream tools: rebuild status.json from the now- # updated manifest. Pollers detect new data via newest_data[]. from . import status - # `deferred` is gone with the Norgate finals gate: no source left here has a - # "not published yet" state — the CFTC zips and the databento raw store are - # either reachable or they are not. The gate itself moved to - # `marketdata-update --bars --require-final`. + # No `deferred` state: the CFTC zips are either reachable or they are not. + # "Not published yet" belongs to the bar producers, and the gate that expresses + # it moved with them (`marketdata-update --bars --require-final`). run = {"kinds": kinds, "failed": failed_kinds, "at": _dt.datetime.utcnow().isoformat(timespec="seconds") + "Z"} path = status.write_status_file(last_run=run) @@ -247,11 +146,11 @@ def main(argv=None, half=None) -> None: def main_cot(argv=None) -> None: - """`cotdata-cot`: the CFTC half. Free, cross-platform, no Norgate.""" - main(argv, half="cot") - - -def main_prices(argv=None) -> None: - """`cotdata-prices`: the price half, now databento only (ADR-0007 moved Norgate - and Yahoo to `marketdata`). Cross-platform; needs DATABENTO_API_KEY to ingest.""" - main(argv, half="prices") + """`cotdata-cot`: an alias for `cotdata-update`, kept because the scheduled jobs + call it by name (see docs/examples/*/run-cot.*). + + It used to be half of a pair. `cotdata-prices` was the other, and each refused the + other's flags so a price box could not quietly become a second COT producer racing + the first. With every price producer moved to marketdata there is no other half to + refuse, so the machinery is gone and only the name survives.""" + main(argv) diff --git a/tests/test_cit_supplemental.py b/tests/test_cit_supplemental.py index 7644fa9..2749c53 100644 --- a/tests/test_cit_supplemental.py +++ b/tests/test_cit_supplemental.py @@ -490,8 +490,17 @@ def test_store_round_trip_and_get_cot(store_env): cot.get_cot("ZW", report="cit") -def test_cotdata_prices_refuses_the_supplemental_action(store_env): - """One host does one job: the price half must not become a second COT producer.""" +def test_the_supplemental_action_is_reachable_from_the_scheduled_entry_point(store_env): + """`cotdata-cot` is what the scheduled job calls, so --cot-supplemental has to be + reachable through it. + + This replaces a test that asserted `cotdata-prices` REFUSED the action, which was + the half-split talking: one host, one job. Every price producer moved to marketdata + (ADR-0007), `cotdata-prices` is gone, and there is no other half to refuse.""" + from unittest import mock + from cotdata import update - with pytest.raises(SystemExit): - update.main_prices(["--cot-supplemental"]) + with mock.patch("cotdata.providers.cftc_cit.update", + return_value={"kind": "cot_supplemental", "ok": True, "wrote": 1}) as m: + update.main_cot(["--cot-supplemental"]) + m.assert_called_once() diff --git a/tests/test_cli_exit.py b/tests/test_cli_exit.py index dc52dbc..e6a0a94 100644 --- a/tests/test_cli_exit.py +++ b/tests/test_cli_exit.py @@ -29,37 +29,16 @@ def test_exits_zero_on_cot_success(tmp_path, monkeypatch): update.main() # must not raise SystemExit -def test_exits_nonzero_when_the_databento_build_fails(tmp_path, monkeypatch): - """The price half is databento only now: ADR-0007 moved the Norgate and Yahoo - producers (and the --require-final finals gate that covered them) to marketdata. - What this file protects is unchanged — a scheduler must be able to tell a failed - run from a quiet one.""" - _argv(monkeypatch, tmp_path, "--build-databento") - from cotdata import update - with mock.patch("cotdata.providers.databento.build", - return_value={"kind": "build_databento", "ok": False, "wrote": 0}): - with pytest.raises(SystemExit) as ei: - update.main() - assert ei.value.code not in (0, None) - - -def test_exits_zero_on_databento_build_success(tmp_path, monkeypatch): - _argv(monkeypatch, tmp_path, "--build-databento") - from cotdata import update - with mock.patch("cotdata.providers.databento.build", - return_value={"kind": "build_databento", "ok": True, "wrote": 3}): - update.main() # must not raise SystemExit - - def test_retired_price_flags_are_refused_not_ignored(tmp_path, monkeypatch): - """A scheduler line still carrying --prices must fail loudly. + """A scheduler line still carrying ANY price action must fail loudly. argparse rejects an unknown flag, so this is really a guard against quietly re-adding one as a no-op alias: a nightly job that keeps exiting 0 while fetching nothing is a store that silently stops being updated. """ from cotdata import update - for flag in ("--prices", "--prices-yahoo", "--metadata", "--require-final"): + for flag in ("--prices", "--prices-yahoo", "--metadata", "--require-final", + "--ingest-databento", "--build-databento", "--reconcile-databento"): _argv(monkeypatch, tmp_path, flag) with pytest.raises(SystemExit) as ei: update.main() diff --git a/tests/test_databento_build.py b/tests/test_databento_build.py deleted file mode 100644 index 757a675..0000000 --- a/tests/test_databento_build.py +++ /dev/null @@ -1,246 +0,0 @@ -"""Stage-2 databento build (ADR-0006): additive back-adjustment from the raw store. - -Populates a raw store directly (as ingest would leave it), runs build(), and reads -the result back out of the store to verify unadj, settlement override, Open Interest, -and the Norgate-style additive back-adjustment. - -Reads via `store.read_prices` rather than a consumer bar API: ADR-0007 moved that API -to `marketdata` and left only the store-level pair databento writes through. The two -differ only in normalisation (`get_prices` sorted the index and named it 'Date'), and -build() writes a sorted DatetimeIndex, so what is asserted here is unchanged. -""" -from pathlib import Path - -import pandas as pd -import pytest - -from cotdata import store -from cotdata.providers.databento import build - - -def _write_ohlcv(raw, symbol, feed, dates, close, sym, instrument_id=None): - idx = pd.DatetimeIndex(pd.to_datetime(dates), name="Date") - n = len(close) - data = {"open": close, "high": [c + 0.5 for c in close], "low": [c - 0.5 for c in close], - "close": close, "volume": [1000] * n, - "symbol": sym if isinstance(sym, list) else [sym] * n} - if instrument_id is not None: - data["instrument_id"] = instrument_id - df = pd.DataFrame(data, index=idx) - p = Path(raw) / "ohlcv" / f"{symbol}{feed}.parquet" - p.parent.mkdir(parents=True, exist_ok=True) - df.to_parquet(p) - - -def _write_stats(raw, symbol, feed, dates, settle=None, oi=None): - idx = pd.to_datetime(dates) - frames = [] - if settle is not None: - frames.append(pd.DataFrame( - {"ts_event": idx, "ts_ref": idx, "stat_type": 3, "price": settle, - "quantity": float("nan")})) - if oi is not None: - frames.append(pd.DataFrame( - {"ts_event": idx, "ts_ref": pd.NaT, "stat_type": 9, "price": float("nan"), - "quantity": oi})) - df = pd.concat(frames, ignore_index=True) - p = Path(raw) / "statistics" / f"{symbol}{feed}.parquet" - p.parent.mkdir(parents=True, exist_ok=True) - df.to_parquet(p) - - -@pytest.fixture -def stores(tmp_path, monkeypatch): - raw, store_dir = tmp_path / "raw", tmp_path / "store" - monkeypatch.setenv("COTDATA_DATABENTO_RAW", str(raw)) - monkeypatch.setenv("COTDATA_STORE", str(store_dir)) - return raw, store_dir - - -def test_build_back_adjusts_the_roll_gap(stores): - raw, _ = stores - dates = pd.date_range("2020-01-01", periods=6, freq="D") - # Front contract A on d1-3, rolls to B at d3's close; B on d4-6. - _write_ohlcv(raw, "ES", ".n.0", dates, [100, 101, 102, 110, 111, 112], ["A", "A", "A", "B", "B", "B"]) - # Second contract carries B on d1-3 (so n1[d3] is B's price the day of the roll). - _write_ohlcv(raw, "ES", ".n.1", dates, [103, 104, 105, 113, 114, 115], ["B", "B", "B", "C", "C", "C"]) - - res = build(["ES"]) - assert res["ok"] and res["wrote"] == 1 - - unadj = store.read_prices("ES", "unadj") - backadj = store.read_prices("ES", "backadj") - - # unadj keeps the raw front prices (roll gap intact: 102 -> 110). - assert list(unadj["Close"]) == [100, 101, 102, 110, 111, 112] - # Roll gap on d3 = n1 - n0 = 105 - 102 = +3, so every price up to & incl. d3 shifts +3. - assert list(backadj["Close"]) == [103, 104, 105, 110, 111, 112] - # The whole bar shifts by the same offset; the newest segment is untouched. - assert backadj["High"].iloc[0] == unadj["High"].iloc[0] + 3 - assert backadj["Low"].iloc[2] == unadj["Low"].iloc[2] + 3 - assert backadj["Close"].iloc[-1] == unadj["Close"].iloc[-1] - - -def test_build_detects_rolls_from_instrument_id_not_symbol(stores): - # The real databento shape: for a continuous series the `symbol` column is a - # CONSTANT alias ("ES.n.0"), and instrument_id is the resolved contract that - # changes at the roll. Roll detection must key on instrument_id, not symbol. - raw, _ = stores - dates = pd.date_range("2020-01-01", periods=6, freq="D") - _write_ohlcv(raw, "ES", ".n.0", dates, [100, 101, 102, 110, 111, 112], - sym="ES.n.0", instrument_id=[10, 10, 10, 20, 20, 20]) - _write_ohlcv(raw, "ES", ".n.1", dates, [103, 104, 105, 113, 114, 115], - sym="ES.n.1", instrument_id=[20, 20, 20, 30, 30, 30]) - - build(["ES"]) - unadj = store.read_prices("ES", "unadj") - backadj = store.read_prices("ES", "backadj") - # The constant `symbol` alias would find no rolls; instrument_id finds the d3 roll, - # gap = 105 - 102 = +3, applied to the pre-roll segment. - assert list(unadj["Close"]) == [100, 101, 102, 110, 111, 112] - assert list(backadj["Close"]) == [103, 104, 105, 110, 111, 112] - - -def test_build_uses_settlement_and_open_interest(stores): - raw, _ = stores - dates = pd.date_range("2020-01-01", periods=6, freq="D") - _write_ohlcv(raw, "ES", ".n.0", dates, [100, 101, 102, 110, 111, 112], ["A", "A", "A", "B", "B", "B"]) - _write_ohlcv(raw, "ES", ".n.1", dates, [103, 104, 105, 113, 114, 115], ["B", "B", "B", "C", "C", "C"]) - # Settlement (stat_type 3) sits 0.5 above the last-trade close; OI (stat_type 9) = 5000. - _write_stats(raw, "ES", ".n.0", dates, settle=[100.5, 101.5, 102.5, 110.5, 111.5, 112.5], oi=[5000] * 6) - _write_stats(raw, "ES", ".n.1", dates, settle=[103.5, 104.5, 105.5, 113.5, 114.5, 115.5]) - - build(["ES"]) - unadj = store.read_prices("ES", "unadj") - backadj = store.read_prices("ES", "backadj") - - # Close is the settlement, not the ohlcv last trade; OI comes from stat_type 9. - assert list(unadj["Close"]) == [100.5, 101.5, 102.5, 110.5, 111.5, 112.5] - assert list(unadj["Open Interest"]) == [5000] * 6 - # Gap measured on settlements: 105.5 - 102.5 = +3.0. - assert list(backadj["Close"]) == [103.5, 104.5, 105.5, 110.5, 111.5, 112.5] - - -def test_build_no_rolls_leaves_series_unadjusted(stores, capsys): - raw, _ = stores - dates = pd.date_range("2020-01-01", periods=5, freq="D") - _write_ohlcv(raw, "ES", ".n.0", dates, [100, 101, 102, 103, 104], ["A"] * 5) - _write_ohlcv(raw, "ES", ".n.1", dates, [110, 111, 112, 113, 114], ["B"] * 5) - - build(["ES"]) - unadj = store.read_prices("ES", "unadj") - backadj = store.read_prices("ES", "backadj") - - assert list(backadj["Close"]) == list(unadj["Close"]) # no roll → no adjustment - assert "no rolls detected" in capsys.readouterr().out - - -def test_build_applies_norgate_unit_scale(stores): - # SI/HG/6J are cents / IMM-x100 in the toolchain's Norgate convention; databento gives - # true dollars. The build scales the price columns x100 so the store is a drop-in, and - # leaves Volume / Open Interest (counts) untouched. - raw, _ = stores - dates = pd.date_range("2020-01-01", periods=4, freq="D") - _write_ohlcv(raw, "SI", ".n.0", dates, [25.0, 25.1, 25.2, 25.3], "A") # $/oz - _write_stats(raw, "SI", ".n.0", dates, oi=[7000] * 4) - - build(["SI"]) - unadj = store.read_prices("SI", "unadj") - assert list(unadj["Close"]) == [2500.0, 2510.0, 2520.0, 2530.0] # x100 -> cents - assert unadj["High"].iloc[0] == (25.0 + 0.5) * 100 - assert list(unadj["Open Interest"]) == [7000] * 4 # counts unscaled - assert list(unadj["Volume"]) == [1000] * 4 - - -def test_build_leaves_unscaled_symbol_in_native_units(stores): - # A symbol not in _PRICE_SCALE (ES) is written as-is — the scale map is a deny-by-default - # allowlist, so a new symbol never gets silently x100'd. - raw, _ = stores - dates = pd.date_range("2020-01-01", periods=3, freq="D") - _write_ohlcv(raw, "ES", ".n.0", dates, [4000.0, 4001.0, 4002.0], "A") - build(["ES"]) - assert list(store.read_prices("ES", "unadj")["Close"]) == [4000.0, 4001.0, 4002.0] - - -def test_build_skips_symbol_missing_from_raw_store(stores): - # Nothing written for CL → build reports it skipped (and it's not 'ok'). - res = build(["CL"]) - assert res["wrote"] == 0 and res["ok"] is False - - -# ── windowed n.1 statistics: same back-adjustment at a fraction of the download ── -class _WResp: - def __init__(self, df): - self._df = df - - def to_df(self): - return self._df - - -class _WClient: - def __init__(self, frames): - self.frames = frames - self.calls = [] - - @property - def timeseries(client_self): - class _TS: - def get_range(ts_self, *, dataset, symbols, stype_in, schema, start, end): - client_self.calls.append((symbols[0], schema, start, end)) - fr = client_self.frames.get((symbols[0], schema)) - if fr is None or fr.empty: - return _WResp(pd.DataFrame()) - naive = fr.index.tz_convert(None) - mask = (naive >= pd.Timestamp(start)) & (naive < pd.Timestamp(end)) - return _WResp(fr[mask]) - return _TS() - - -def _roll_frames(dates): - idx = pd.to_datetime(dates).tz_localize("UTC") - idx.name = "ts_event" - - def ohlcv(close, iid): - return pd.DataFrame({"open": close, "high": close, "low": close, "close": close, - "volume": [1000] * len(close), "instrument_id": iid, - "symbol": ["ES.n"] * len(close)}, index=idx) - - def stats(settle): - return pd.DataFrame({"ts_ref": idx, "stat_type": 3, "price": settle, - "quantity": float("nan")}, index=idx) - - return { - ("ES.n.0", "ohlcv-1d"): ohlcv([100, 101, 102, 110, 111, 112], [10, 10, 10, 20, 20, 20]), - ("ES.n.1", "ohlcv-1d"): ohlcv([103, 104, 105, 113, 114, 115], [20, 20, 20, 30, 30, 30]), - ("ES.n.0", "statistics"): stats([100.5, 101.5, 102.5, 110.5, 111.5, 112.5]), - ("ES.n.1", "statistics"): stats([103.5, 104.5, 105.5, 113.5, 114.5, 115.5]), - } - - -def test_windowed_n1_stats_matches_full_backadj(tmp_path, monkeypatch): - from cotdata import store - from cotdata.providers.databento import ingest - dates = pd.date_range("2020-01-01", periods=6) # roll at 2020-01-03 (id 10→20) - - def run(tag, window): - root = tmp_path / tag - monkeypatch.setenv("COTDATA_DATABENTO_RAW", str(root / "raw")) - monkeypatch.setenv("COTDATA_STORE", str(root / "store")) - client = _WClient(_roll_frames(dates)) - ingest(symbols=["ES"], client=client, end="2020-01-07", cold_start="2020-01-01", - n1_stats_window=window) - build(["ES"]) - return store.read_prices("ES", "backadj"), client - - full_bad, full_client = run("full", None) - win_bad, win_client = run("win", 1) - - # Identical back-adjustment (gap = n1_settle - n0_settle = 105.5 - 102.5 = +3 either way). - assert list(win_bad["Close"]) == list(full_bad["Close"]) == [103.5, 104.5, 105.5, 110.5, 111.5, 112.5] - - # ...but the n.1 statistics fetch was a narrow roll window, not the full range. - def n1_stat_spans(c): - return [(s, e) for (sym, sch, s, e) in c.calls if sym == "ES.n.1" and sch == "statistics"] - assert n1_stat_spans(full_client) == [("2020-01-01", "2020-01-07")] # one full-range pull - win_spans = n1_stat_spans(win_client) - assert win_spans and all(s >= "2020-01-02" and e <= "2020-01-05" for s, e in win_spans) diff --git a/tests/test_databento_ingest.py b/tests/test_databento_ingest.py deleted file mode 100644 index 8ac55de..0000000 --- a/tests/test_databento_ingest.py +++ /dev/null @@ -1,514 +0,0 @@ -"""Stage-1 databento ingest (ADR-0006): raw landing store + resume manifest. - -Exercised with an injected fake client shaped like ``databento.Historical`` — no -API key, no network. Verifies the raw files, the fetched-range manifest, the -resume-from-last_date behaviour, and that databento-null symbols are skipped. -""" -import json - -import pandas as pd -import pytest - -from cotdata.providers.databento import _ingest_manifest_path, ingest, reconcile_manifest - - -# ── a databento.Historical-shaped fake ─────────────────────────────────────── -class _FakeResp: - def __init__(self, df): - self._df = df - - def to_df(self): - return self._df - - -class _FakeTS: - def __init__(self, owner): - self.owner = owner - - def get_range(self, *, dataset, symbols, stype_in, schema, start, end): - self.owner.calls.append((symbols[0], schema, start, end)) - frame = self.owner.frames.get((symbols[0], schema)) - if frame is None or frame.empty: - return _FakeResp(pd.DataFrame()) - s, e = pd.Timestamp(start), pd.Timestamp(end) - # databento returns ts_event as the index for both schemas. - naive = frame.index.tz_convert(None) - mask = (naive >= s) & (naive <= e) - return _FakeResp(frame[mask]) - - -class _FakeClient: - def __init__(self, frames): - self.frames = frames - self.calls = [] - - @property - def timeseries(self): - return _FakeTS(self) - - -def _ohlcv(dates, base): - idx = pd.to_datetime(dates).tz_localize("UTC") - idx.name = "ts_event" - n = len(idx) - return pd.DataFrame( - {"open": [base] * n, "high": [base + 1] * n, "low": [base - 1] * n, - "close": [base + 0.5] * n, "volume": [1000] * n, "symbol": ["ES.FUT"] * n}, - index=idx) - - -def _stats(dates, price): - idx = pd.to_datetime(dates).tz_localize("UTC") - idx.name = "ts_event" - n = len(idx) - return pd.DataFrame( - {"ts_ref": idx, "stat_type": [3] * n, "price": [price] * n, "quantity": [0] * n}, - index=idx) - - -def _frames(dates): - return { - ("ES.n.0", "ohlcv-1d"): _ohlcv(dates, 100), - ("ES.n.1", "ohlcv-1d"): _ohlcv(dates, 101), - ("ES.n.0", "statistics"): _stats(dates, 100.5), - ("ES.n.1", "statistics"): _stats(dates, 101.5), - } - - -# ── tests ───────────────────────────────────────────────────────────────────── -def test_ingest_writes_raw_files_and_manifest(tmp_path, monkeypatch): - monkeypatch.setenv("COTDATA_DATABENTO_RAW", str(tmp_path)) - dates = pd.date_range("2020-01-01", periods=5, freq="D") - client = _FakeClient(_frames(dates)) - - res = ingest(symbols=["ES"], client=client, end="2020-01-05", cold_start="2020-01-01") - - assert res["ok"] and res["symbols"] == 1 - for feed in (".n.0", ".n.1"): - assert (tmp_path / "ohlcv" / f"ES{feed}.parquet").exists() - assert (tmp_path / "statistics" / f"ES{feed}.parquet").exists() - - ohlcv = pd.read_parquet(tmp_path / "ohlcv" / "ES.n.0.parquet") - assert len(ohlcv) == 5 - assert ohlcv.index.tz is None # bronze is tz-naive - assert ohlcv.index.is_monotonic_increasing - - man = json.loads((tmp_path / "ingest_manifest.json").read_text()) - assert man["ES.n.0:ohlcv-1d"]["last_date"] == "2020-01-05" - assert man["ES.n.0:ohlcv-1d"]["first_date"] == "2020-01-01" - assert man["ES.n.0:ohlcv-1d"]["n_rows"] == 5 - - -def test_ingest_pages_statistics_into_year_chunks(tmp_path, monkeypatch): - # A from-inception statistics pull is paged into _STATS_CHUNK_DAYS windows so no single - # get_range is large enough to time out; ohlcv (tiny) stays one request. Shrink the - # window so a short fixture range still pages. - import cotdata.providers.databento as dbmod - monkeypatch.setenv("COTDATA_DATABENTO_RAW", str(tmp_path)) - monkeypatch.setattr(dbmod, "_STATS_CHUNK_DAYS", 2) - dates = pd.date_range("2020-01-01", periods=6) - client = _FakeClient(_frames(dates)) - - dbmod.ingest(symbols=["ES"], client=client, end="2020-01-06", cold_start="2020-01-01") - - stat_calls = [c for c in client.calls if c[0] == "ES.n.0" and c[1] == "statistics"] - ohlcv_calls = [c for c in client.calls if c[0] == "ES.n.0" and c[1] == "ohlcv-1d"] - assert len(stat_calls) >= 3 # 5-day span paged by 2-day windows - assert len(ohlcv_calls) == 1 # ohlcv is not paged - # Overlapping fake boundaries notwithstanding, the raw store dedupes to one row per day. - combined = pd.read_parquet(tmp_path / "statistics" / "ES.n.0.parquet") - assert len(combined) == 6 - man = json.loads((tmp_path / "ingest_manifest.json").read_text()) - assert man["ES.n.0:statistics"]["last_date"] == "2020-01-06" - - -class _FlakyStatsTS(_FakeTS): - """get_range that raises on the statistics window starting on ``fail_start`` until the - owner's ``heal`` flag flips — models a mid-history page failure that a re-run recovers.""" - def get_range(self, *, dataset, symbols, stype_in, schema, start, end): - if (schema == "statistics" and start == self.owner.fail_start - and not self.owner.heal): - raise TimeoutError("read timed out") - return super().get_range(dataset=dataset, symbols=symbols, stype_in=stype_in, - schema=schema, start=start, end=end) - - -class _FlakyStatsClient(_FakeClient): - def __init__(self, frames, fail_start): - super().__init__(frames) - self.fail_start, self.heal = fail_start, False - - @property - def timeseries(self): - return _FlakyStatsTS(self) - - -def test_ingest_resumes_mid_history_after_a_page_failure(tmp_path, monkeypatch): - # A page failure part-way through a paged statistics pull must leave the manifest at the - # last good page, so a re-run resumes there rather than re-pulling from inception. - import cotdata.providers.databento as dbmod - monkeypatch.setenv("COTDATA_DATABENTO_RAW", str(tmp_path)) - monkeypatch.setattr(dbmod, "_STATS_CHUNK_DAYS", 2) - dates = pd.date_range("2020-01-01", periods=8) - client = _FlakyStatsClient(_frames(dates), fail_start="2020-01-05") - - dbmod.ingest(symbols=["ES"], client=client, end="2020-01-08", cold_start="2020-01-01") - man = json.loads((tmp_path / "ingest_manifest.json").read_text()) - # The windows before 2020-01-05 landed; the 2020-01-05 window failed, so the watermark - # stops before the end rather than being advanced past the gap. - assert man["ES.n.0:statistics"]["last_date"] < "2020-01-08" - - # Re-run with the flake healed: resume from the failed window, finish the pull. - client.heal = True - client.calls.clear() - dbmod.ingest(symbols=["ES"], client=client, end="2020-01-08", cold_start="2020-01-01") - resumed = [c for c in client.calls if c[0] == "ES.n.0" and c[1] == "statistics"] - assert resumed and resumed[0][2] >= "2020-01-04" # did NOT restart from 2020-01-01 - man = json.loads((tmp_path / "ingest_manifest.json").read_text()) - assert man["ES.n.0:statistics"]["last_date"] == "2020-01-08" - combined = pd.read_parquet(tmp_path / "statistics" / "ES.n.0.parquet") - assert len(combined) == 8 - - -def test_ingest_resumes_from_last_date(tmp_path, monkeypatch): - monkeypatch.setenv("COTDATA_DATABENTO_RAW", str(tmp_path)) - - # First pull: 5 days. - ingest(symbols=["ES"], client=_FakeClient(_frames(pd.date_range("2020-01-01", periods=5))), - end="2020-01-05", cold_start="2020-01-01") - - # Second pull: source now has 8 days; a fresh client so we can inspect its calls. - client2 = _FakeClient(_frames(pd.date_range("2020-01-01", periods=8))) - ingest(symbols=["ES"], client=client2, end="2020-01-08", cold_start="2020-01-01") - - # Resume: the ohlcv .n.0 call must start the day AFTER the stored last_date. - ohlcv_calls = [c for c in client2.calls if c[0] == "ES.n.0" and c[1] == "ohlcv-1d"] - assert ohlcv_calls and ohlcv_calls[0][2] == "2020-01-06" - - combined = pd.read_parquet(tmp_path / "ohlcv" / "ES.n.0.parquet") - assert len(combined) == 8 # 5 + 3 appended, no dupes - man = json.loads((tmp_path / "ingest_manifest.json").read_text()) - assert man["ES.n.0:ohlcv-1d"]["last_date"] == "2020-01-08" - assert man["ES.n.0:ohlcv-1d"]["n_rows"] == 8 - - -def test_ingest_noop_when_already_current(tmp_path, monkeypatch): - monkeypatch.setenv("COTDATA_DATABENTO_RAW", str(tmp_path)) - ingest(symbols=["ES"], client=_FakeClient(_frames(pd.date_range("2020-01-01", periods=5))), - end="2020-01-05", cold_start="2020-01-01") - - client2 = _FakeClient(_frames(pd.date_range("2020-01-01", periods=5))) - ingest(symbols=["ES"], client=client2, end="2020-01-05", cold_start="2020-01-01") - assert client2.calls == [] # start would be > end → nothing fetched - - -def test_ingest_skips_databento_null_symbol(tmp_path, monkeypatch): - monkeypatch.setenv("COTDATA_DATABENTO_RAW", str(tmp_path)) - client = _FakeClient({}) - res = ingest(symbols=["CC"], client=client, end="2020-01-05") # CC is databento: null - assert res["symbols"] == 0 - assert client.calls == [] - - -def test_ingest_skips_when_start_equals_end(tmp_path, monkeypatch): - # Regression: an up-to-date symbol computes start == end, which databento rejects - # (422 data_time_range_start_on_or_after_end). The guard must skip, not fetch. - monkeypatch.setenv("COTDATA_DATABENTO_RAW", str(tmp_path)) - ingest(symbols=["ES"], client=_FakeClient(_frames(pd.date_range("2020-01-01", periods=5))), - end="2020-01-05", cold_start="2020-01-01") # stores last_date = 2020-01-05 - # end == stored last_date + 1 → start (last+1) == end → must skip, no API call. - client2 = _FakeClient(_frames(pd.date_range("2020-01-01", periods=6))) - ingest(symbols=["ES"], client=client2, end="2020-01-06", cold_start="2020-01-01") - assert client2.calls == [] - - -def test_fetch_retries_transient_failures_then_succeeds(): - from cotdata.providers.databento import _fetch - n = {"calls": 0} - - class _TS: - def get_range(self, **k): - n["calls"] += 1 - if n["calls"] < 3: - raise ConnectionError("Response ended prematurely") - return _FakeResp(pd.DataFrame({"x": [1]})) - - class _C: - timeseries = _TS() - - df = _fetch(_C(), "GLBX.MDP3", "ES.n.0", "ohlcv-1d", "2020-01-01", "2020-01-05", - retries=3, backoff=0) - assert n["calls"] == 3 and not df.empty - - -def test_fetch_raises_after_exhausting_retries(): - from cotdata.providers.databento import _fetch - - class _TS: - def get_range(self, **k): - raise TimeoutError("read timed out") - - class _C: - timeseries = _TS() - - with pytest.raises(TimeoutError): - _fetch(_C(), "GLBX.MDP3", "ES.n.0", "ohlcv-1d", "2020-01-01", "2020-01-05", - retries=2, backoff=0) - - -class _FakeBatch: - """databento client.batch stand-in: submit → 'done' → download CSV files.""" - def __init__(self, dates): - self.dates = dates - self.submitted = [] - - def submit_job(self, **k): - self.submitted.append(k) - return {"id": "j1", "state": "queued"} - - def list_jobs(self): - return [{"id": "j1", "state": "done"}] - - def download(self, *, job_id, output_dir): - import os - k = self.submitted[-1] - schema, syms = k["schema"], k["symbols"] - rows = [] - for sym in syms: - for i, d in enumerate(self.dates): - base = {"ts_event": d.isoformat(), "instrument_id": 10, "symbol": sym} - if schema == "ohlcv-1d": - rows.append({**base, "open": 100 + i, "high": 100 + i, "low": 100 + i, - "close": 100 + i, "volume": 1000}) - else: - rows.append({**base, "ts_ref": d.isoformat(), "stat_type": 3, - "price": 100.5 + i, "quantity": 0}) - p = os.path.join(output_dir, f"out.{schema}.csv") - pd.DataFrame(rows).to_csv(p, index=False) - return [p] - - -class _FakeBatchClient: - def __init__(self, dates): - self.batch = _FakeBatch(dates) - - -def test_ingest_batch_writes_raw_and_manifest(tmp_path, monkeypatch): - monkeypatch.setenv("COTDATA_DATABENTO_RAW", str(tmp_path)) - dates = pd.date_range("2020-01-01", periods=4) - from cotdata.providers.databento import ingest_batch - - res = ingest_batch(symbols=["ES"], client=_FakeBatchClient(dates), - end="2020-01-05", cold_start="2020-01-01") - assert res["ok"] and res["symbols"] == 1 - - man = json.loads((tmp_path / "ingest_manifest.json").read_text()) - assert man["ES.n.0:ohlcv-1d"]["last_date"] == "2020-01-04" - assert man["ES.n.0:ohlcv-1d"]["batch"] is True - assert man["ES.n.0:statistics"]["last_date"] == "2020-01-04" - - ohlcv = pd.read_parquet(tmp_path / "ohlcv" / "ES.n.0.parquet") - assert len(ohlcv) == 4 and ohlcv.index.name == "Date" and ohlcv.index.tz is None - stats = pd.read_parquet(tmp_path / "statistics" / "ES.n.0.parquet") - assert (stats["stat_type"] == 3).all() - - -def test_reconcile_manifest_rebuilds_from_raw_store(tmp_path, monkeypatch): - monkeypatch.setenv("COTDATA_DATABENTO_RAW", str(tmp_path)) - dates = pd.date_range("2020-01-01", periods=5) - ingest(symbols=["ES"], client=_FakeClient(_frames(dates)), end="2020-01-05", cold_start="2020-01-01") - - # Simulate a run that wrote the raw parquets but never persisted the manifest. - _ingest_manifest_path().unlink() - - recorded = reconcile_manifest() - assert recorded, "reconcile should record the on-disk raw tables" - man = json.loads(_ingest_manifest_path().read_text()) - assert man["ES.n.0:ohlcv-1d"]["last_date"] == "2020-01-05" - assert man["ES.n.0:ohlcv-1d"]["n_rows"] == 5 - assert man["ES.n.0:ohlcv-1d"]["reconciled"] is True - assert man["ES.n.0:statistics"]["last_date"] == "2020-01-05" - - # With the manifest rebuilt, a re-ingest sees everything current -> no API calls. - client2 = _FakeClient(_frames(dates)) - ingest(symbols=["ES"], client=client2, end="2020-01-05", cold_start="2020-01-01") - assert client2.calls == [] - - -# ── reconcile: the manifest must match the disk in BOTH directions ──────── -def _raw_layout(tmp_path, monkeypatch): - monkeypatch.setenv("COTDATA_STORE", str(tmp_path)) - root = tmp_path / "_raw" / "databento" - (root / "ohlcv").mkdir(parents=True) - (root / "statistics").mkdir(parents=True) - return root - - -def _write_raw(root, sub, name, dates): - """ts_event is the INDEX for both schemas, matching what _append_raw writes.""" - import pandas as pd - idx = pd.DatetimeIndex(pd.to_datetime(dates), name="ts_event") - df = pd.DataFrame({"open": range(len(dates)), "high": range(len(dates)), - "low": range(len(dates)), "close": range(len(dates)), - "volume": range(len(dates))}, index=idx) - df.to_parquet(root / sub / f"{name}.parquet") - - -def test_reconcile_prunes_entries_whose_parquet_is_missing(tmp_path, monkeypatch): - """The dangerous direction. ingest() derives each table's start from last_date and - skips it when already current, without ever checking the file exists. A manifest - entry with no file therefore means that table is NEVER fetched again: no error, no - rows, a permanent hole in a paid dataset.""" - import json - - from cotdata.providers import databento - root = _raw_layout(tmp_path, monkeypatch) - _write_raw(root, "ohlcv", "ES.n.0", ["2026-07-01", "2026-07-02"]) - (root / "ingest_manifest.json").write_text(json.dumps({ - "ES.n.0:ohlcv-1d": {"last_date": "2026-07-02", "n_rows": 2}, - "GC.n.0:ohlcv-1d": {"last_date": "2026-07-02", "n_rows": 99}, # no file - "GC.n.0:statistics": {"last_date": "2026-07-02", "n_rows": 99}, # no file - })) - - res = databento.reconcile_manifest() - assert set(res["pruned"]) == {"GC.n.0:ohlcv-1d", "GC.n.0:statistics"} - - m = json.loads((root / "ingest_manifest.json").read_text()) - assert "ES.n.0:ohlcv-1d" in m # has a file, kept - assert "GC.n.0:ohlcv-1d" not in m # pruned, so the next run re-fetches - - -def test_reconcile_still_records_files_missing_from_the_manifest(tmp_path, monkeypatch): - """The original direction: a run interrupted before persisting the manifest.""" - import json - - from cotdata.providers import databento - root = _raw_layout(tmp_path, monkeypatch) - _write_raw(root, "ohlcv", "ES.n.0", ["2026-07-01", "2026-07-02"]) - (root / "ingest_manifest.json").write_text(json.dumps({})) - - res = databento.reconcile_manifest() - assert "ES.n.0:ohlcv-1d" in res["recorded"] - assert res["pruned"] == [] - - -def test_reconcile_prune_can_be_turned_off(tmp_path, monkeypatch): - import json - - from cotdata.providers import databento - root = _raw_layout(tmp_path, monkeypatch) - (root / "ingest_manifest.json").write_text(json.dumps( - {"GC.n.0:ohlcv-1d": {"last_date": "2026-07-02", "n_rows": 99}})) - - res = databento.reconcile_manifest(prune=False) - assert res["pruned"] == [] - assert "GC.n.0:ohlcv-1d" in json.loads((root / "ingest_manifest.json").read_text()) - - -# ── advance markers are not ghosts ─────────────────────────────────────────── -# A paged pull that meets an empty window writes {last_date: ...} with NO parquet, on -# purpose, so a re-run does not refetch that empty span forever. For a symbol whose -# data starts after the GLBX floor, that marker is all there is. Pruning on "no file" -# alone deletes exactly those and restores the loop they exist to prevent, every run, -# against a paid API. - -def test_an_empty_window_marker_survives_the_prune(tmp_path, monkeypatch): - import json - - from cotdata.providers import databento - root = _raw_layout(tmp_path, monkeypatch) - _write_raw(root, "ohlcv", "ES.n.0", ["2026-07-01"]) - (root / "ingest_manifest.json").write_text(json.dumps({ - "ES.n.0:ohlcv-1d": {"last_date": "2026-07-01", "n_rows": 1}, - # The marker: last_date only, no first_date, no n_rows, no file. - "BTC.n.0:statistics": {"last_date": "2020-12-31"}, - })) - - res = databento.reconcile_manifest() - - assert res["pruned"] == [], "an advance marker is not a ghost" - m = json.loads((root / "ingest_manifest.json").read_text()) - assert m["BTC.n.0:statistics"]["last_date"] == "2020-12-31", "marker was destroyed" - - -def test_a_real_ghost_beside_a_marker_is_still_pruned(tmp_path, monkeypatch): - """The narrowing must not cost the prune its purpose.""" - import json - - from cotdata.providers import databento - root = _raw_layout(tmp_path, monkeypatch) - (root / "ingest_manifest.json").write_text(json.dumps({ - "BTC.n.0:statistics": {"last_date": "2020-12-31"}, # marker - "GC.n.0:ohlcv-1d": {"last_date": "2026-07-02", "n_rows": 99, - "first_date": "2020-01-01"}, # ghost - })) - - res = databento.reconcile_manifest() - - assert res["pruned"] == ["GC.n.0:ohlcv-1d"] - m = json.loads((root / "ingest_manifest.json").read_text()) - assert "BTC.n.0:statistics" in m and "GC.n.0:ohlcv-1d" not in m - - -def test_a_zero_row_entry_is_treated_as_a_marker(tmp_path, monkeypatch): - """n_rows == 0 asserts no data, so there is nothing for a missing file to contradict.""" - import json - - from cotdata.providers import databento - root = _raw_layout(tmp_path, monkeypatch) - (root / "ingest_manifest.json").write_text(json.dumps({ - "BTC.n.0:statistics": {"last_date": "2020-12-31", "n_rows": 0}, - })) - - assert databento.reconcile_manifest()["pruned"] == [] - - -def test_an_unrecognised_entry_shape_errs_toward_keeping(tmp_path, monkeypatch): - """A wrongly-KEPT entry costs one skipped table that --ingest-databento reports. A - wrongly-PRUNED marker costs a paid refetch on every run after. Fail the cheap way.""" - import json - - from cotdata.providers import databento - root = _raw_layout(tmp_path, monkeypatch) - (root / "ingest_manifest.json").write_text(json.dumps({ - "BTC.n.0:statistics": {"last_date": "2020-12-31", "n_rows": "lots"}, - "ETH.n.0:statistics": {"windowed": True, "batch": True, "reconciled": True}, - })) - - assert databento.reconcile_manifest()["pruned"] == [] - - -def test_the_marker_the_real_ingest_writes_survives_reconcile(tmp_path, monkeypatch): - """End to end against the actual writer, not a hand-built manifest. - - The GLBX-floor case: a symbol whose data begins after the whole requested range, so - EVERY window is empty and the `elif chunk_days` branch is the only thing that ever - writes. The result is a manifest of pure advance markers and no parquet directory - at all. Under a "no file means ghost" rule reconcile deletes both entries and the - next ingest re-scans the empty span, on every run, against a paid API. - - A first version of this test used data that started INSIDE the range. That produced - a marker on the first window which the next window immediately overwrote with real - rows, so the assertion had nothing to assert and the test skipped. A skipped test is - not evidence. - """ - import cotdata.providers.databento as dbmod - monkeypatch.setenv("COTDATA_DATABENTO_RAW", str(tmp_path)) - monkeypatch.setattr(dbmod, "_STATS_CHUNK_DAYS", 2) - client = _FakeClient(_frames(pd.date_range("2021-06-01", periods=3))) - - dbmod.ingest(symbols=["ES"], client=client, end="2020-01-06", - cold_start="2020-01-01") - - man_path = tmp_path / "ingest_manifest.json" - markers = {k: v for k, v in json.loads(man_path.read_text()).items() - if not v.get("n_rows")} - assert markers, "fixture no longer produces an advance marker; the test is vacuous" - assert not (tmp_path / "ohlcv").exists(), "expected no parquet for an all-empty pull" - - dbmod.reconcile_manifest() - - after = json.loads(man_path.read_text()) - for key, val in markers.items(): - assert key in after, f"reconcile destroyed the advance marker {key}" - assert after[key]["last_date"] == val["last_date"] diff --git a/tests/test_databento_provider.py b/tests/test_databento_provider.py deleted file mode 100644 index e8403c7..0000000 --- a/tests/test_databento_provider.py +++ /dev/null @@ -1,193 +0,0 @@ -"""Smoke tests for the DORMANT Databento provider. It isn't in the live EOD path, -but it carries the hard-won statistics logic (Open Interest = stat_type 9; -settlement = stat_type 3, NOT 7=LOWEST_OFFER, dated by ts_ref) that the intraday -work will reuse — so lock that parsing here. `databento` is an optional extra, so -mock the SDK (as test_norgate_provider mocks norgatedata) to run on any machine.""" -import sys -import types -from unittest import mock - -import pandas as pd -import pytest - -# databento is optional (the [databento] extra); force a mock module so the lazy -# `import databento as db` inside the provider resolves without the SDK installed. -mock_databento = types.ModuleType("databento") -sys.modules["databento"] = mock_databento - -from cotdata.providers import databento as dbprov # noqa: E402,I001 (import after sys.modules mock injection above) - - -def _client(ohlcv_df, stats_df): - """A fake databento Historical whose timeseries.get_range returns the OHLCV - frame for schema 'ohlcv-1d' and the statistics frame otherwise.""" - client = mock.Mock() - - def get_range(**kwargs): - res = mock.Mock() - res.to_df.return_value = ohlcv_df if kwargs.get("schema") == "ohlcv-1d" else stats_df - return res - - client.timeseries.get_range.side_effect = get_range - return client - - -@pytest.fixture -def ohlcv(): - idx = pd.DatetimeIndex(["2020-01-02"], name="ts_event") - return pd.DataFrame( - {"open": [99.0], "high": [101.0], "low": [98.0], "close": [100.0], "volume": [1234]}, - index=idx, - ) - - -@pytest.fixture -def stats(): - # OI (stat_type 9) is disseminated for the session date; the settlement - # (stat_type 3) for the 01-02 session is disseminated the NEXT morning - # (ts_event 01-03) but carries ts_ref 01-02 — the session it applies to. - idx = pd.DatetimeIndex(["2020-01-02", "2020-01-03"], name="ts_event") - return pd.DataFrame( - { - "stat_type": [9, 3], - "price": [0.0, 101.5], - "quantity": [5000.0, 0.0], - "ts_ref": pd.to_datetime(["2020-01-02", "2020-01-02"]), - }, - index=idx, - ) - - -def test_provider_imports_without_sdk(): - """The provider module imports and exposes its entry points even though - `databento` isn't a hard dependency (import is lazy, behind the extra).""" - assert callable(dbprov.fetch_daily_ohlc) - assert callable(dbprov.update_all_daily_prices) - - -def test_fetch_extracts_open_interest_from_stat_type_9(tmp_path, monkeypatch, ohlcv, stats): - monkeypatch.setenv("COTDATA_STORE", str(tmp_path)) - monkeypatch.setenv("DATABENTO_API_KEY", "test-key") - mock_databento.Historical = mock.Mock(return_value=_client(ohlcv, stats)) - - df = dbprov.fetch_daily_ohlc("ES", price_type="close") - - assert df.loc["2020-01-02", "Close"] == 100.0 # ohlcv close, untouched - assert df.loc["2020-01-02", "Open Interest"] == 5000.0 # from stat_type 9 - - -def test_settlement_overrides_close_dated_by_ts_ref(tmp_path, monkeypatch, ohlcv, stats): - """price_type='settlement' replaces Close with stat_type 3's price, joined by - ts_ref (the session), so the next-morning-disseminated settle lands on 01-02.""" - monkeypatch.setenv("COTDATA_STORE", str(tmp_path)) - monkeypatch.setenv("DATABENTO_API_KEY", "test-key") - mock_databento.Historical = mock.Mock(return_value=_client(ohlcv, stats)) - - df = dbprov.fetch_daily_ohlc("ES", price_type="settlement") - - assert df.loc["2020-01-02", "Close"] == 101.5 # settle, not ohlcv 100.0 - assert df.loc["2020-01-02", "Open Interest"] == 5000.0 - - -# ── start_date ────────────────────────────────────────────────────────────── -# start_date means two different things depending on cache state (see the -# fetch_daily_ohlc docstring): it narrows the FETCH on a cold cache (cost benefit — -# the whole reason this was broken), but only narrows the RETURN on a warm cache -# (correctness — never lets a later, narrower caller truncate what other callers -# already rely on being cached). Each get is exercised separately below. - -def test_cold_cache_start_date_narrows_the_fetch_floor(tmp_path, monkeypatch, ohlcv, stats): - monkeypatch.setenv("COTDATA_STORE", str(tmp_path)) - monkeypatch.setenv("DATABENTO_API_KEY", "test-key") - client = _client(ohlcv, stats) - mock_databento.Historical = mock.Mock(return_value=client) - - dbprov.fetch_daily_ohlc("ES", start_date="2019-01-01", price_type="close") - - ohlcv_call = next(c for c in client.timeseries.get_range.call_args_list - if c.kwargs.get("schema") == "ohlcv-1d") - assert ohlcv_call.kwargs["start"] == "2019-01-01" # NOT "2000-01-01" - - -def test_cold_cache_start_date_is_still_clamped_to_the_glbx_floor(tmp_path, monkeypatch, - ohlcv, stats): - monkeypatch.setenv("COTDATA_STORE", str(tmp_path)) - monkeypatch.setenv("DATABENTO_API_KEY", "test-key") - client = _client(ohlcv, stats) - mock_databento.Historical = mock.Mock(return_value=client) - - dbprov.fetch_daily_ohlc("ES", start_date="1990-01-01", price_type="close") - - ohlcv_call = next(c for c in client.timeseries.get_range.call_args_list - if c.kwargs.get("schema") == "ohlcv-1d") - assert ohlcv_call.kwargs["start"] == "2010-06-06" # clamped, not 1990 - - -def test_returned_frame_excludes_rows_before_start_date(tmp_path, monkeypatch, ohlcv, stats): - monkeypatch.setenv("COTDATA_STORE", str(tmp_path)) - monkeypatch.setenv("DATABENTO_API_KEY", "test-key") - mock_databento.Historical = mock.Mock(return_value=_client(ohlcv, stats)) - - df = dbprov.fetch_daily_ohlc("ES", start_date="2020-01-03", price_type="close") - - assert "2020-01-02" not in df.index.strftime("%Y-%m-%d") # before start_date - - -def test_no_start_date_is_unbounded_default(tmp_path, monkeypatch, ohlcv, stats): - """The default (None) is unchanged from before this parameter existed.""" - monkeypatch.setenv("COTDATA_STORE", str(tmp_path)) - monkeypatch.setenv("DATABENTO_API_KEY", "test-key") - client = _client(ohlcv, stats) - mock_databento.Historical = mock.Mock(return_value=client) - - df = dbprov.fetch_daily_ohlc("ES", price_type="close") - - ohlcv_call = next(c for c in client.timeseries.get_range.call_args_list - if c.kwargs.get("schema") == "ohlcv-1d") - assert ohlcv_call.kwargs["start"] == "2010-06-06" # 2000-01-01 clamped to GLBX floor - assert "2020-01-02" in df.index.strftime("%Y-%m-%d") - - -def test_warm_cache_start_date_does_not_narrow_the_fetch_but_still_filters_the_return( - tmp_path, monkeypatch): - """The case the docstring warns about: a cache already holding 06-01..06-03 must - keep being topped up from its own last_date, unaffected by a later, narrower - start_date — but the RETURN must still respect it, excluding 06-01.""" - monkeypatch.setenv("COTDATA_STORE", str(tmp_path)) - monkeypatch.setenv("DATABENTO_API_KEY", "test-key") - monkeypatch.setattr(dbprov, "_API_LAST_CHECKED", {}) # fresh throttle state - - cache_dir = tmp_path / "_cache" / "databento" - cache_dir.mkdir(parents=True) - existing = pd.DataFrame( - {"Open": [10.0, 11.0, 12.0], "High": [10.5, 11.5, 12.5], - "Low": [9.5, 10.5, 11.5], "Close": [10.2, 11.2, 12.2], - "Volume": [100, 100, 100], "Open Interest": [500.0, 500.0, 500.0]}, - index=pd.DatetimeIndex(["2019-06-01", "2019-06-02", "2019-06-03"], name="Date")) - existing.to_parquet(cache_dir / "ES_daily.parquet") - - new_ohlcv = pd.DataFrame( - {"open": [13.0], "high": [13.5], "low": [12.5], "close": [13.2], "volume": [100]}, - index=pd.DatetimeIndex(["2019-06-04"], name="ts_event")) - new_stats = pd.DataFrame( - {"stat_type": [9], "price": [0.0], "quantity": [600.0], - "ts_ref": pd.to_datetime(["2019-06-04"])}, - index=pd.DatetimeIndex(["2019-06-04"], name="ts_event")) - client = _client(new_ohlcv, new_stats) - mock_databento.Historical = mock.Mock(return_value=client) - - df = dbprov.fetch_daily_ohlc("ES", start_date="2019-06-02", force_refresh=False, - price_type="close") - - ohlcv_call = next(c for c in client.timeseries.get_range.call_args_list - if c.kwargs.get("schema") == "ohlcv-1d") - assert ohlcv_call.kwargs["start"] == "2019-06-04" # resumed from last_date+1, - # NOT narrowed to start_date - dates = set(df.index.strftime("%Y-%m-%d")) - assert "2019-06-01" not in dates # excluded: before start_date - assert {"2019-06-02", "2019-06-03", "2019-06-04"} <= dates - - # the ON-DISK cache still holds the full series — start_date shaped the - # return, not what was persisted. - on_disk = pd.read_parquet(cache_dir / "ES_daily.parquet") - assert "2019-06-01" in on_disk.index.strftime("%Y-%m-%d") diff --git a/tests/test_manifest_seam.py b/tests/test_manifest_seam.py index 5293921..1c63ac9 100644 --- a/tests/test_manifest_seam.py +++ b/tests/test_manifest_seam.py @@ -1,12 +1,19 @@ -"""The COT / price seam: one manifest per producer half (ADR-0007 step 1). - -`_touch_manifest` is a read-modify-write. Two producers sharing one manifest.json -eventually lose an entry, and cotdata has two producers by design: the CFTC downloader -(any OS) and the price producer (Windows for Norgate). Splitting the manifest by half -means they never touch the same file. - -Nothing writes the legacy aggregate any more. It is read only as a per-half fallback -for a store that has not run `--migrate-manifests` yet. +"""The manifest seam, and what is left of it now that one half is empty. + +ADR-0007 step 1 split the manifest per producer half, because `_touch_manifest` is a +read-modify-write and two producers sharing one file eventually lose an entry. cotdata +had two producers by design: the CFTC downloader and the price producer. + +**It has one now.** Every price producer moved to marketdata, so nothing writes the +`prices` half any more, and the `cotdata-prices` entry point that enforced the split is +gone with it. What survives is the part these tests exist for: a REAL store on disk +still carries `prices` and `metadata` entries from before the move, and the code has to +keep reading, migrating and reconciling them rather than stranding them. So `prices` +appears throughout this file as LEGACY data written by hand, never through a writer — +there is no writer. + +Nothing writes the legacy aggregate `manifest.json` either. It is read only as a +per-domain fallback for a store that has not run `--migrate-manifests` yet. """ import json @@ -20,10 +27,19 @@ def store_env(tmp_path, monkeypatch): return tmp_path -def _prices(): - idx = pd.date_range("2020-01-01", periods=3, freq="D", name="Date") - return pd.DataFrame({"Open": [1, 2, 3], "High": [2, 3, 4], "Low": [0, 1, 2], - "Close": [1.5, 2.5, 3.5]}, index=idx) +def _legacy_prices_half(root, entries=None): + """Write a `manifests/prices.json` by hand, as a pre-ADR-0007 store carries it. + + By hand because there is no writer any more — which is the condition under test. + """ + import json as _json + path = root / "manifests" / "prices.json" + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(_json.dumps({ + "schema_version": 2, + "prices": entries if entries is not None else {"ES_backadj": {"n_rows": 3}}, + })) + return path def _cot(): @@ -46,7 +62,12 @@ def test_an_undeclared_domain_refuses_to_write(): store.half_for("options_oi") -def test_price_and_cot_domains_land_on_opposite_sides(): +def test_the_retired_price_domains_are_still_declared(): + """`prices` and `metadata` have no writer left, and are still on the map on + purpose: `migrate_manifests` SKIPS an undeclared domain, so undeclaring them would + strand a real store's existing entries in the legacy aggregate forever, and + `reconcile_manifest` would resolve their directory by fallback instead of by + declaration.""" from cotdata import store assert store.half_for("prices") == "prices" assert store.half_for("metadata") == "prices" @@ -54,27 +75,38 @@ def test_price_and_cot_domains_land_on_opposite_sides(): assert store.half_for("cot_tff") == "cot" +def test_nothing_can_write_the_price_half_any_more(): + """The other side of the same coin: declared for reading, with no way to write.""" + from cotdata import store + for gone in ("write_prices", "read_prices", "write_metadata", "read_metadata"): + assert not hasattr(store, gone), gone + + # ── writes ──────────────────────────────────────────────────────────────── -def test_the_two_halves_do_not_share_a_file(store_env): +def test_a_cot_write_never_touches_the_price_half(store_env): + """The split's remaining job. The price half is now a file only a previous version + wrote, so the COT producer must leave it exactly alone — a read-modify-write that + reached across would destroy data no producer can regenerate here.""" from cotdata import config, store - store.write_prices("ES", "backadj", _prices(), source="test") + legacy = _legacy_prices_half(store_env) + before = legacy.read_text() + store.write_cot_legacy("ES_13874A", _cot(), source="test") - prices_half = json.loads(config.manifest_path_for("prices").read_text()) cot_half = json.loads(config.manifest_path_for("cot").read_text()) - assert "prices" in prices_half and "cot_legacy" not in prices_half assert "cot_legacy" in cot_half and "prices" not in cot_half + assert legacy.read_text() == before # byte-for-byte untouched def test_a_clobbered_legacy_file_cannot_lose_an_entry(store_env): - """The hazard, simulated. A concurrent producer (or a file-level sync between two - stores) overwrites the shared aggregate wholesale. Both entries must survive, - because a migrated store never consults that file.""" + """The hazard, simulated. A file-level sync between two stores overwrites the + shared aggregate wholesale. Both entries must survive, because a migrated store + never consults that file.""" from cotdata import config, store - store.write_prices("ES", "backadj", _prices(), source="test") + _legacy_prices_half(store_env) store.write_cot_legacy("ES_13874A", _cot(), source="test") - # A racing producer rewrites the aggregate from its own stale copy. + # A stale sync rewrites the aggregate from an older copy. config.manifest_path().write_text(json.dumps({"schema_version": 2, "prices": {}})) m = store.load_manifest() @@ -98,28 +130,32 @@ def test_half_data_wins_over_stale_legacy(store_env): from cotdata import config, store config.manifest_path().parent.mkdir(parents=True, exist_ok=True) config.manifest_path().write_text(json.dumps( - {"schema_version": 2, "prices": {"ES_backadj": {"n_rows": 1}}})) - store.write_prices("ES", "backadj", _prices(), source="test") - assert store.load_manifest()["prices"]["ES_backadj"]["n_rows"] == 3 + {"schema_version": 2, "cot_legacy": {"ES_13874A": {"n_rows": 1}}})) + store.write_cot_legacy("ES_13874A", _cot(), source="test") + assert store.load_manifest()["cot_legacy"]["ES_13874A"]["n_rows"] == 2 def test_a_half_that_never_ran_still_resolves_from_legacy(store_env): - """Mid-transition: the price producer has run on the new code, the COT one has not.""" + """The real shape of an un-migrated store today: the COT producer has run on the + new code, and the prices left behind by the old one are only in the aggregate.""" from cotdata import config, store config.manifest_path().parent.mkdir(parents=True, exist_ok=True) config.manifest_path().write_text(json.dumps( - {"schema_version": 2, "cot_legacy": {"GC_088691": {"n_rows": 5}}})) - store.write_prices("ES", "backadj", _prices(), source="test") + {"schema_version": 2, "prices": {"GC_backadj": {"n_rows": 5}}})) + store.write_cot_legacy("ES_13874A", _cot(), source="test") m = store.load_manifest() - assert m["prices"]["ES_backadj"]["n_rows"] == 3 # from the half - assert m["cot_legacy"]["GC_088691"]["n_rows"] == 5 # from legacy + assert m["cot_legacy"]["ES_13874A"]["n_rows"] == 2 # from the half + assert m["prices"]["GC_backadj"]["n_rows"] == 5 # from legacy def test_empty_store_returns_the_empty_shape(store_env): + """COT domains only. A store created from here on never gets a `prices` key, + because nothing here can write one.""" from cotdata import store m = store.load_manifest() - assert m["prices"] == {} and m["cot_legacy"] == {} + assert m["cot_legacy"] == {} and m["cot_supplemental"] == {} + assert "prices" not in m and "metadata" not in m assert m["schema_version"] >= 1 @@ -130,69 +166,40 @@ def test_manifest_path_for_rejects_an_unknown_half(store_env): def test_status_check_reads_the_merged_manifest(store_env): - """--check works off the manifest only, so it must see per-half writes.""" + """--check works off the manifest only, so it must see both a live COT write and + the legacy price entries a real store still carries.""" from cotdata import status, store - store.write_prices("ES", "backadj", _prices(), source="test") + _legacy_prices_half(store_env) store.write_cot_legacy("ES_13874A", _cot(), source="test") s = status.summarize(store.load_manifest()) assert "prices" in s["domains"] and "cot_legacy" in s["domains"] -# ── half-scoped CLI entry points ────────────────────────────────────────── -def test_cot_entry_point_refuses_price_actions(store_env, capsys): - """A price host that could also run --cot-all becomes a second COT producer - racing the first, which is exactly what the split manifests exist to contain.""" - from cotdata import update - with pytest.raises(SystemExit): - update.main_cot(["--build-databento"]) - assert "belong(s) to the prices half" in capsys.readouterr().err - - -def test_prices_entry_point_refuses_cot_actions(store_env, capsys): +# ── the entry points that are left ──────────────────────────────────────── +def test_cotdata_cot_is_an_alias_and_not_a_scoped_half(store_env): + """`cotdata-cot` survives because the scheduled jobs call it by name, but it no + longer SCOPES anything: there is no other half to refuse. It must behave exactly + like `cotdata-update`, so a wrapper script keeps working unchanged.""" from cotdata import update - with pytest.raises(SystemExit): - update.main_prices(["--cot-all"]) - assert "belong(s) to the cot half" in capsys.readouterr().err + assert not hasattr(update, "main_prices") # the price half is gone + assert not hasattr(update, "_reject_other_half") + assert not hasattr(update, "_HALF_ACTIONS") -def test_read_only_actions_work_from_either_half(store_env): - from cotdata import store, update - store.write_prices("ES", "backadj", _prices(), source="test") - update.main_cot(["--check"]) - update.main_prices(["--check"]) + update.main_cot(["--check"]) # read-only, no network + update.main(["--check"]) -def test_combined_entry_point_applies_no_half_restriction(): - """cotdata-update keeps working for a single-machine deployment, so passing both - halves' actions must not be rejected the way the scoped entry points do.""" - import argparse - - from cotdata import update - parser = argparse.ArgumentParser() - both = argparse.Namespace(build_databento=True, cot_all=True, - ingest_databento=False, cot_legacy=False, - cot_disagg=False, cot_tff=False) - # Each scoped half rejects the other's action ... - for half in ("cot", "prices"): - with pytest.raises(SystemExit): - update._reject_other_half(parser, both, half) - # ... and main() only calls it when a half is set, which is what leaves - # cotdata-update unrestricted. - assert "half" in update.main.__code__.co_varnames - - -def test_every_action_flag_is_assigned_to_a_half(): - """A new action must be classified, or the entry points silently allow it. - - Read off the PARSER, not a list copied beside it. The copied list was the bug: - it stayed green through this change while naming three flags that no longer - exist, so it could not have caught a fourth being added either. Now a new flag - fails here until it is put on one side of the seam or declared a non-action. - """ +def test_the_retired_price_flags_are_refused_not_ignored(store_env): + """A scheduler line still carrying a price action must fail loudly. argparse + rejects an unknown flag, so this really guards against re-adding one as a silent + no-op: a nightly job that keeps exiting 0 while fetching nothing is a store that + quietly stops being updated.""" from cotdata import update - flags = {a.dest for a in update._parser()._actions} - update._NON_ACTIONS - assigned = set(update._HALF_ACTIONS["cot"]) | set(update._HALF_ACTIONS["prices"]) - assert flags == assigned + for flag in ("--prices", "--metadata", "--ingest-databento", "--build-databento"): + with pytest.raises(SystemExit) as ei: + update.main([flag]) + assert ei.value.code not in (0, None), flag # ── dropping the legacy aggregate ───────────────────────────────────────── @@ -201,8 +208,8 @@ def test_writes_no_longer_touch_the_legacy_aggregate(store_env): lose each other's entries, and a file-level sync between two stores resolves it last-writer-wins. Nothing writes it any more.""" from cotdata import config, store - store.write_prices("ES", "backadj", _prices(), source="test") - assert config.manifest_path_for("prices").exists() + store.write_cot_legacy("ES_13874A", _cot(), source="test") + assert config.manifest_path_for("cot").exists() assert not config.manifest_path().exists() @@ -227,12 +234,12 @@ def test_migrate_splits_a_legacy_manifest(store_env): def test_migrate_is_idempotent_and_never_resurrects_stale_entries(store_env): from cotdata import config, store - store.write_prices("ES", "backadj", _prices(), source="test") # 3 rows, current + store.write_cot_legacy("ES_13874A", _cot(), source="test") # 2 rows, current config.manifest_path().write_text(json.dumps( - {"schema_version": 2, "prices": {"ES_backadj": {"n_rows": 999}}})) # stale + {"schema_version": 2, "cot_legacy": {"ES_13874A": {"n_rows": 999}}})) # stale assert store.migrate_manifests() == {"prices": 0, "cot": 0} - assert store.load_manifest()["prices"]["ES_backadj"]["n_rows"] == 3 + assert store.load_manifest()["cot_legacy"]["ES_13874A"]["n_rows"] == 2 assert store.migrate_manifests() == {"prices": 0, "cot": 0} # re-run is a no-op @@ -240,7 +247,10 @@ def test_an_incomplete_half_file_still_falls_back_per_domain(store_env): """Found on a real store. manifests/prices.json held `prices` but not `metadata`, because the price producer had run on the new code while the metadata producer had not. A per-HALF fallback rule hid `metadata` entirely until the migration - ran, so the fallback is per DOMAIN.""" + ran, so the fallback is per DOMAIN. + + Still exercised with the price domains, because that is still the shape a real + store has: a half file holding one retired domain and not the other.""" from cotdata import config, store config.manifest_path().parent.mkdir(parents=True, exist_ok=True) config.manifest_path().write_text(json.dumps({ @@ -248,7 +258,7 @@ def test_an_incomplete_half_file_still_falls_back_per_domain(store_env): "prices": {"OLD_backadj": {"n_rows": 1}}, "metadata": {"contract_specs": {"n_rows": 47}}, })) - store.write_prices("ES", "backadj", _prices(), source="test") # prices half only + _legacy_prices_half(store_env) # prices half only, no metadata m = store.load_manifest() assert "ES_backadj" in m["prices"] # from the half file @@ -256,8 +266,9 @@ def test_an_incomplete_half_file_still_falls_back_per_domain(store_env): def test_legacy_is_read_only_for_a_domain_that_has_not_migrated(store_env): - """A store can be part migrated: prices written by the new code, COT still only - in the aggregate.""" + """A store can be part migrated: one domain moved to its half file, another still + only in the aggregate. Once a domain has a half file the aggregate is not consulted + for it at all, so a stale entry there cannot come back.""" from cotdata import config, store config.manifest_path().parent.mkdir(parents=True, exist_ok=True) config.manifest_path().write_text(json.dumps({ @@ -265,7 +276,7 @@ def test_legacy_is_read_only_for_a_domain_that_has_not_migrated(store_env): "prices": {"OLD_backadj": {"n_rows": 1}}, "cot_legacy": {"GC_088691": {"n_rows": 5}}, })) - store.write_prices("ES", "backadj", _prices(), source="test") + _legacy_prices_half(store_env) # prices migrated, cot not m = store.load_manifest() assert m["cot_legacy"]["GC_088691"]["n_rows"] == 5 # from legacy, cot unmigrated @@ -275,7 +286,7 @@ def test_legacy_is_read_only_for_a_domain_that_has_not_migrated(store_env): def test_fully_migrated_store_ignores_the_legacy_file(store_env): from cotdata import config, store - store.write_prices("ES", "backadj", _prices(), source="test") + _legacy_prices_half(store_env) store.write_cot_legacy("GC_088691", _cot(), source="test") config.manifest_path().write_text(json.dumps( {"schema_version": 2, "prices": {"GHOST": {"n_rows": 1}}, diff --git a/tests/test_registry.py b/tests/test_registry.py index 9987c02..c402c9d 100644 --- a/tests/test_registry.py +++ b/tests/test_registry.py @@ -6,10 +6,8 @@ from cotdata.registry import ( all_symbols, by_asset_class, - default_price_source, hist_code_scales, load_registry, - resolve_source, symbol, ) @@ -23,23 +21,31 @@ def test_registry_loads_all_symbols(): def test_basic_symbol_loading(): es = symbol("ES") assert es.internal == "ES" - assert es.norgate == "&ES" assert es.asset_class == "Equities" assert es.is_equity is True assert es.cftc_code == "13874A" assert es.hist_codes == () -def test_yahoo_only_market_has_no_norgate_symbol(): - """MSCI MME/MFS are priced off ETF proxies (EEM/EFA); Norgate carries no series - for them, so norgate is None (registry `norgate: null`) — that's the signal the - Norgate producer filters on. A defaulted '&MME' would send it fetching a - nonexistent &MME_CCB.""" - for s in ("MME", "MFS"): - assert symbol(s).norgate is None, s - assert symbol(s).yahoo in ("EEM", "EFA") - # Norgate-covered markets still default to '&'. - assert symbol("ES").norgate == "&ES" +def test_the_registry_carries_no_vendor_mappings(): + """ADR-0007: this is a CFTC-positioning registry. The norgate / yahoo / databento + columns, the `price_source` override and `resolve_source` moved to marketdata's + registry with the producers that read them. + + Asserted rather than assumed, because re-adding a column is one line and a stale + vendor mapping here would be read as authoritative by whoever finds it — while the + thing that actually resolves vendors lives in the other package and would have + moved on. + """ + import dataclasses + + from cotdata import registry as reg + + fields = {f.name for f in dataclasses.fields(symbol("ES"))} + assert fields == {"internal", "asset_class", "is_equity", "report_type", + "cftc_code", "hist_codes"} + for gone in ("PRICE_SOURCES", "resolve_source", "default_price_source", "_can_serve"): + assert not hasattr(reg, gone), gone def test_symbol_with_simple_hist_codes(): @@ -94,84 +100,6 @@ def test_is_equity_derived_from_asset_class(): # ── price-source selection: capability + deployment default + override ──────── -def test_databento_mapping_defaults_to_internal_root(): - # Norgate-covered CME/CBOT/NYMEX/COMEX markets default databento to the root. - assert symbol("ES").databento == "ES" - assert symbol("CL").databento == "CL" - assert symbol("GC").databento == "GC" - - -def test_databento_null_for_non_glbx_markets(): - # ICE softs + lumber (not on GLBX Globex) and MSCI intl are databento: null. - for s in ("SB", "CT", "CC", "KC", "OJ", "LBR", "MME", "MFS"): - assert symbol(s).databento is None, s - - -def test_resolve_source_uses_deployment_default_when_capable(): - assert resolve_source(symbol("ES"), "norgate") == "norgate" - assert resolve_source(symbol("ES"), "databento") == "databento" - assert resolve_source(symbol("CL"), "databento") == "databento" - - -def test_resolve_source_falls_back_to_yfinance_when_default_cannot_serve(): - # MME/MFS have neither a norgate nor a databento series, but do have a yahoo - # ETF proxy — so either deployment default resolves to yfinance. - assert resolve_source(symbol("MME"), "norgate") == "yfinance" - assert resolve_source(symbol("MME"), "databento") == "yfinance" - - -def test_softs_have_yahoo_fallback_and_resolve_to_yfinance_on_databento(): - # ICE softs aren't on GLBX but carry a Yahoo continuous fallback, so a databento - # deployment resolves them to yfinance; Norgate still covers them locally. - for s in ("SB", "CT", "CC", "KC", "OJ", "LBR"): - assert symbol(s).yahoo, s - assert resolve_source(symbol(s), "databento") == "yfinance", s - assert resolve_source(symbol(s), "norgate") == "norgate", s - - -def test_resolve_source_none_when_no_vendor_can_serve(tmp_path): - # A synthetic market with no vendor mapping at all → nothing can price it. - reg = load_registry(_write(tmp_path, textwrap.dedent(""" - Metals: - XX: - cftc_code: "000000" - norgate: null - databento: null - """))) - assert resolve_source(reg["XX"], "databento") is None - assert resolve_source(reg["XX"], "norgate") is None - - -def test_price_source_override_wins(tmp_path): - reg = load_registry(_write(tmp_path, textwrap.dedent(""" - Metals: - GC: - cftc_code: "088691" - price_source: yfinance - yahoo: "GLD" - """))) - assert reg["GC"].price_source == "yfinance" - # Override beats the deployment default even though Norgate could serve GC. - assert resolve_source(reg["GC"], "norgate") == "yfinance" - - -def test_invalid_price_source_override_raises(tmp_path): - with pytest.raises(ValueError, match="price_source"): - load_registry(_write( - tmp_path, 'Metals:\n GC:\n cftc_code: "1"\n price_source: bloomberg\n')) - - -def test_default_price_source_env(monkeypatch): - monkeypatch.delenv("COTDATA_PRICE_SOURCE", raising=False) - assert default_price_source() == "norgate" - monkeypatch.setenv("COTDATA_PRICE_SOURCE", "databento") - assert default_price_source() == "databento" - monkeypatch.setenv("COTDATA_PRICE_SOURCE", "nope") - with pytest.raises(ValueError, match="COTDATA_PRICE_SOURCE"): - default_price_source() - - -# ── $COTDATA_REGISTRY / explicit-path override (the point of the refactor) ──── _MINI_YAML = textwrap.dedent(""" Metals: GC: @@ -230,7 +158,7 @@ def test_duplicate_symbol_raises(tmp_path): def test_missing_cftc_code_raises(tmp_path): with pytest.raises(ValueError, match="missing cftc_code"): - load_registry(_write(tmp_path, "Metals:\n GC:\n norgate: \"&GC\"\n")) + load_registry(_write(tmp_path, "Metals:\n GC:\n is_equity: false\n")) def test_scalar_attrs_raises(tmp_path): diff --git a/tests/test_status.py b/tests/test_status.py index b6ea64b..8fcc565 100644 --- a/tests/test_status.py +++ b/tests/test_status.py @@ -127,18 +127,17 @@ def test_write_status_file_roundtrip(tmp_path, monkeypatch): from cotdata import status as st from cotdata import store # seed the store manifest via a real write - idx = __import__("pandas").date_range("2026-07-10", periods=3, freq="D", name="Date") - df = __import__("pandas").DataFrame({"Open": [1, 2, 3], "High": [1, 2, 3], "Low": [1, 2, 3], - "Close": [1, 2, 3], "Volume": [1, 2, 3], - "Open Interest": [1, 2, 3]}, index=idx) - store.write_prices("ES", "backadj", df, source="test") + idx = __import__("pandas").date_range("2026-07-10", periods=3, freq="D", + name="Report_Date") + df = __import__("pandas").DataFrame({"Open_Interest_All": [1, 2, 3]}, index=idx) + store.write_cot_legacy("ES_13874A", df, source="test") - path = st.write_status_file(last_run={"kinds": ["prices"], "ok": ["ES"], "failed": []}) + path = st.write_status_file(last_run={"kinds": ["cot_legacy"], "ok": ["ES"], "failed": []}) assert path.endswith("status.json") doc = json.loads((tmp_path / "status.json").read_text()) - assert doc["newest_data"]["prices"] == "2026-07-12" - assert doc["last_run"]["kinds"] == ["prices"] - assert doc["domains"]["prices"]["entries"] == 1 + assert doc["newest_data"]["cot_legacy"] == "2026-07-12" + assert doc["last_run"]["kinds"] == ["cot_legacy"] + assert doc["domains"]["cot_legacy"]["entries"] == 1 def test_run_summary_ok_and_failed(): diff --git a/tests/test_store.py b/tests/test_store.py index 7e9a0dc..e3c9e90 100644 --- a/tests/test_store.py +++ b/tests/test_store.py @@ -20,24 +20,28 @@ def _sample(): }, index=idx) -def test_prices_roundtrip_and_manifest(store_env): - """The store-level price pair, which the databento producer still writes through. +def _cot_sample(): + idx = pd.date_range("2020-01-07", periods=3, freq="W-TUE", name="Report_Date") + return pd.DataFrame({"Open_Interest_All": [100, 110, 120], + "NonComm_Positions_Long_All": [10, 12, 14]}, index=idx) - The CONSUMER bar API (get_prices/roll_dates/propadj, the volume views) left with - ADR-0007 and is tested in marketdata now — see test_consumer_bar_api_is_gone. - """ + +def test_cot_roundtrip_and_manifest(store_env): + """A write lands its parquet AND its manifest entry, with the provenance a consumer + reads. The price equivalent of this test is in marketdata now: every bar producer, + and the store-level pair they wrote through, left with ADR-0007.""" from cotdata import load_manifest, store - store.write_prices("ES", "backadj", _sample(), source="test") + store.write_cot_legacy("ES_13874A", _cot_sample(), source="test") - df = store.read_prices("ES", "backadj") - assert list(df.columns)[:6] == ["Open", "High", "Low", "Close", "Volume", "Open Interest"] - assert len(df) == 5 - assert store.read_prices("ZZ", "backadj").empty # absent symbol -> empty + df = store.read_cot_legacy("ES_13874A") + assert list(df.columns) == ["Open_Interest_All", "NonComm_Positions_Long_All"] + assert len(df) == 3 + assert store.read_cot_legacy("ZZ_000000").empty # absent table -> empty m = load_manifest() - assert m["prices"]["ES_backadj"]["n_rows"] == 5 - assert m["prices"]["ES_backadj"]["source"] == "test" - assert m["prices"]["ES_backadj"]["last_date"] == "2020-01-05" + assert m["cot_legacy"]["ES_13874A"]["n_rows"] == 3 + assert m["cot_legacy"]["ES_13874A"]["source"] == "test" + assert m["cot_legacy"]["ES_13874A"]["last_date"] == "2020-01-21" def test_reconcile_prunes_ghosts_keeps_real(store_env): @@ -84,22 +88,25 @@ def test_consumer_bar_api_is_gone(store_env): for name in ("get_prices", "roll_dates"): assert not hasattr(cotdata, name) assert name not in cotdata.__all__ - for mod in ("cotdata.prices", "cotdata.providers.norgate", "cotdata.providers.yfinance"): + for mod in ("cotdata.prices", "cotdata.providers.norgate", + "cotdata.providers.yfinance", "cotdata.providers.databento"): with pytest.raises(ModuleNotFoundError): importlib.import_module(mod) -def test_contract_specs_are_no_longer_written_here(store_env): - """Contract specs moved to marketdata with the bars they describe (§7.2). - - The domain stays DECLARED so pre-ADR-0007 stores still migrate and reconcile - their `metadata` entries instead of stranding them — but there is no writer. +def test_no_price_surface_of_any_kind_survives(store_env): + """Contract specs went with the bars (§7.2), and the databento producer that was + the last writer of `prices/` followed them. Both domains stay DECLARED so a + pre-ADR-0007 store still migrates and reconciles its entries instead of stranding + them — but there is no writer, and no reader, for either. """ from cotdata import store - assert store.half_for("metadata") == "prices" # still mapped, for old stores - for name in ("write_metadata", "upsert_metadata", "read_metadata"): - assert not hasattr(store, name) + for domain in ("prices", "metadata"): + assert store.half_for(domain) == "prices" # still mapped, for old stores + for name in ("write_prices", "read_prices", + "write_metadata", "upsert_metadata", "read_metadata"): + assert not hasattr(store, name), name def test_schema_version_and_require_schema(store_env): @@ -109,7 +116,7 @@ def test_schema_version_and_require_schema(store_env): # Empty store → no manifest yet → load_manifest defaults to config.SCHEMA_VERSION assert schema_version() == cfg.SCHEMA_VERSION - store.write_prices("ES", "backadj", _sample(), source="test") + store.write_cot_legacy("ES_13874A", _cot_sample(), source="test") assert schema_version() == cfg.SCHEMA_VERSION # stamped by the write require_schema(cfg.SCHEMA_VERSION) # satisfied → no raise with pytest.raises(RuntimeError): diff --git a/tests/test_validate_databento.py b/tests/test_validate_databento.py deleted file mode 100644 index 6e01c3c..0000000 --- a/tests/test_validate_databento.py +++ /dev/null @@ -1,113 +0,0 @@ -"""Unit tests for the ADR-0006 item-6 validation harness (scripts/validate_databento_vs_norgate.py). - -The harness itself needs real Norgate + databento stores to run; here we verify its -comparison logic against synthetic backadj frames: that an anchor-only difference -passes, a scale (unit) mismatch fails, per-day outliers are counted, roll dates are -read from Delivery Month, and read_backadj round-trips a store parquet. -""" -import importlib.util -from pathlib import Path - -import numpy as np -import pandas as pd -import pytest - -_SCRIPT = Path(__file__).resolve().parents[1] / "scripts" / "validate_databento_vs_norgate.py" -_spec = importlib.util.spec_from_file_location("validate_databento_vs_norgate", _SCRIPT) -val = importlib.util.module_from_spec(_spec) -_spec.loader.exec_module(val) - - -_DATES = pd.date_range("2020-01-01", periods=250, freq="B") -_RNG = np.random.default_rng(0) -_CHANGES = _RNG.normal(0, 1.0, len(_DATES)) -_NG_CLOSE = 100 + np.cumsum(_CHANGES) - - -def _frame(close, dm=None): - df = pd.DataFrame({"Open": close, "High": close, "Low": close, "Close": close}, - index=pd.DatetimeIndex(_DATES, name="Date")) - if dm is not None: - df["Delivery Month"] = dm - return df - - -def _pass_thresholds(m): - return val.evaluate(m, corr_min=0.999, scale_band=0.02, rel_tol=0.10, max_outliers_per_yr=8) - - -def test_anchor_only_difference_passes(): - # databento = norgate + a constant (a floating back-adjust anchor) → identical - # daily changes, different level. This is the expected clean case. - ng = _frame(_NG_CLOSE) - db = _frame(_NG_CLOSE + 7.5) - m = val.compare(ng, db, "ES") - - assert m["change_corr"] > 0.99999 - assert m["scale_ratio"] == pytest.approx(1.0, abs=1e-9) - assert m["change_max_rel_diff"] < 1e-9 - assert m["level_mean_abs"] == pytest.approx(7.5) - assert _pass_thresholds(m) == [] - - -def test_scale_mismatch_fails(): - # databento daily changes are 1.5x norgate's (e.g. a settlement/unit scale bug): - # perfectly correlated but the wrong size → caught by scale_ratio, not corr. - ng = _frame(_NG_CLOSE) - db = _frame(100 + np.cumsum(_CHANGES * 1.5)) - m = val.compare(ng, db, "ES") - - assert m["change_corr"] > 0.999 # still correlated - assert m["scale_ratio"] == pytest.approx(1.5, abs=0.05) - fails = _pass_thresholds(m) - assert any("scale_ratio" in f for f in fails) - - -def test_per_day_outliers_are_counted_and_gate_fails(): - # Inject a 4-point discrepancy every 12th day (~20/yr, each ~4x a typical daily - # move) — the kind of thing a roll-date mismatch or a bad settlement would cause. - ng = _frame(_NG_CLOSE) - db_changes = _CHANGES.copy() - db_changes[::12] += 4.0 - db = _frame(100 + np.cumsum(db_changes)) - m = val.compare(ng, db, "ES") - - assert m["outlier_days"] >= 15 - assert m["change_max_rel_diff"] > 1.0 # worst day dwarfs a typical move - assert _pass_thresholds(m) != [] # too many outlier days → fail - # Loosening the outlier budget past what occurred clears that specific gate. - loose = val.evaluate(m, corr_min=0.0, scale_band=1.0, rel_tol=0.10, max_outliers_per_yr=1000) - assert loose == [] - - -def test_insufficient_overlap_reports_error(): - ng = _frame(_NG_CLOSE).iloc[:2] - db = _frame(_NG_CLOSE).iloc[100:102] # no common dates - m = val.compare(ng, db, "ES") - assert m.get("error") == "insufficient overlap" - assert _pass_thresholds(m) == ["insufficient overlap"] - - -def test_roll_dates_from_delivery_month(): - dm = ["A"] * 100 + ["B"] * 150 - f = _frame(_NG_CLOSE, dm=dm) - rolls = val._roll_dates(f) - assert len(rolls) == 1 and rolls[0] == _DATES[100] - # roll counts surface in the comparison metrics - m = val.compare(f, _frame(_NG_CLOSE + 1, dm=dm), "ES") - assert m["rolls_norgate"] == 1 and m["rolls_common"] == 1 - - -@pytest.mark.parametrize("layout", ["prices", "bars/futures/norgate"]) -def test_read_backadj_roundtrip(tmp_path, layout): - """Both store layouts, because this harness now spans two packages: ADR-0007 - moved the Norgate side into marketdata (`bars///`) while the - databento side still writes cotdata's flat `prices/`. Reading only one would - silently report the Norgate store as absent and skip every symbol — a green - run that compared nothing.""" - d = tmp_path / layout - d.mkdir(parents=True) - _frame(_NG_CLOSE).to_parquet(d / "ES_backadj.parquet") - got = val.read_backadj(str(tmp_path), "ES") - assert not got.empty and got.index.name == "Date" and got.index.tz is None - assert val.read_backadj(str(tmp_path), "NOPE").empty