diff --git a/.github/workflows/python-test.yml b/.github/workflows/python-test.yml index 1438abc..919b751 100644 --- a/.github/workflows/python-test.yml +++ b/.github/workflows/python-test.yml @@ -30,6 +30,15 @@ jobs: repository: mspinola/cotdata path: cotdata + - name: Check out marketdata (daily bars — resolves `-e ../marketdata`) + # ADR-0007 moved bars out of cotdata, and marketdata is not on PyPI, so this + # sibling checkout is the only thing that resolves the dependency cotmetrics + # now imports. + uses: actions/checkout@v4 + with: + repository: mspinola/marketdata + path: marketdata + - name: Check out cotmetrics (metrics/data layer — resolves `-e ../cotmetrics`) uses: actions/checkout@v4 with: @@ -48,7 +57,7 @@ jobs: python -m pip install -r requirements.txt - name: Verify internal dependency floors - # cotdata / cotmetrics are installed editable from ../ at whatever HEAD was + # cotdata / marketdata / cotmetrics are installed editable from ../ at whatever HEAD was # checked out, so pip never enforces the version floors in pyproject.toml. # This asserts the installed siblings actually satisfy those declared floors. working-directory: cot-analyzer @@ -62,9 +71,12 @@ jobs: working-directory: cot-analyzer env: PYTHONPATH: src - # Dummy store so cotdata's COTDATA_STORE guard doesn't error at import. + # Dummy stores so each package's store guard doesn't error at import. + # Two of them since ADR-0007: positioning and bars are separate roots, and + # neither package defaults a missing root to somewhere plausible. COTDATA_STORE: /tmp/cotdata_store + MARKETDATA_STORE: /tmp/marketdata_store COTMETRICS_CACHE: /tmp/cotmetrics_cache run: | - mkdir -p /tmp/cotdata_store /tmp/cotmetrics_cache + mkdir -p /tmp/cotdata_store /tmp/marketdata_store /tmp/cotmetrics_cache pytest tests/ diff --git a/pyproject.toml b/pyproject.toml index 0f96d73..8def06a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -33,6 +33,7 @@ dependencies = [ "pyyaml==6.0.3", "requests==2.32.5", "cotdata>=0.1.0", + "marketdata>=0.1.0", # bars; ADR-0007 moved them out of cotdata "cotmetrics[options]>=0.3.0", ] diff --git a/requirements.txt b/requirements.txt index f2c5443..da701a8 100644 --- a/requirements.txt +++ b/requirements.txt @@ -18,10 +18,12 @@ ruff==0.15.22 # Linter — pinned so CI installs it and stays reproduc tomli==2.0.1; python_version < "3.11" # for scripts/check_dep_floors.py on <3.11 # data/metrics layer (editable siblings). cotmetrics provides the COT index, -# signals, indexer + the legacy ETL/options/scheduler; it depends on cotdata. -# Both are published on PyPI (cotdata, cotmetrics), but we install them EDITABLE +# signals, indexer + the legacy ETL/options/scheduler; it depends on cotdata for +# CFTC positioning and, since ADR-0007 moved bars out, on marketdata for prices. +# All three are published on PyPI, but we install them EDITABLE # from sibling checkouts so in-workspace changes are picked up immediately and CI # catches integration breaks before those siblings are released. Pin the PyPI # releases instead if you want reproducible, checkout-free builds. -e ../cotdata +-e ../marketdata -e ../cotmetrics[options] diff --git a/scripts/check_dep_floors.py b/scripts/check_dep_floors.py index f0db7ca..e5956cb 100644 --- a/scripts/check_dep_floors.py +++ b/scripts/check_dep_floors.py @@ -31,6 +31,7 @@ # rather than a package index, so only these need the extra check. INTERNAL = { "cotdata", + "marketdata", "cotmetrics", "crucible", "crucible-stack", diff --git a/server-side/README.md b/server-side/README.md index af4f902..cdc434e 100644 --- a/server-side/README.md +++ b/server-side/README.md @@ -140,12 +140,23 @@ chmod 600 /root/trading_workspace/cot-analyzer/.env **Required:** ```bash -COTDATA_STORE=/root/cotdata_store # the synced price store, see step 6 +COTDATA_STORE=/root/cotdata_store # synced CFTC positioning store, see step 6 +MARKETDATA_STORE=/root/marketdata_store # synced daily-bar store, see step 6 ``` Nothing works without `COTDATA_STORE`. Every entry point resolves instruments through it, so a missing or empty store fails at import rather than degrading. +**`MARKETDATA_STORE` is required too, and it is new.** ADR-0007 makes `cotdata` CFTC +positioning only and moves every bar to `marketdata`, so the price reads behind the +indexer, the signal rejection scores and the options max-pain snapshot now resolve +against a second store. It fails the same way the first one does — by name, at the +point of read — rather than serving an empty frame that reads as "no data for this +instrument". + +Both are **synced**, not produced here. This box cannot produce bars at all: +`norgatedata` drives a locally installed Norgate Data Updater and NDU is Windows-only. + **Required for the emailed Signal Matrix report:** ```bash @@ -215,8 +226,11 @@ They are cotmetrics runtime state, and PR #12 moved them out of this repo into `.local-state/cot-analyzer/`, so the deprecated script could not have shipped them anyway. **The server rebuilds the cache itself** — confirmed by the maintainer on 2026-08-04, not inferred from the push having been broken. `CotIndexer` writes per-instrument parquet under -`COTMETRICS_CACHE` on first use and busts it on the upstream `cotdata.schema_version()` plus -`METRICS_CACHE_VERSION`, so a store push is the only input it needs. +`COTMETRICS_CACHE` on first use and busts it on BOTH upstream store versions +(`cotdata.schema_version()` and `marketdata.schema_version()`) plus +`METRICS_CACHE_VERSION`, so a store push is the only input it needs. Both are watched +because the case that guard was written for — reconstructed volume being promoted — was +a *price* schema bump, and prices now live in the second store. The first sync after a release is therefore slower than steady state, because the cache is cold. That is a latency cost on one request, not a missing payload. @@ -226,8 +240,15 @@ The store push by hand, for reference, is: ```bash rsync -avz --no-o --no-g --progress --exclude='.DS_Store' \ /path/to/cotdata_store/ USER@HOST:/root/cotdata_store/ + +rsync -avz --no-o --no-g --progress --exclude='.DS_Store' \ + /path/to/marketdata_store/ USER@HOST:/root/marketdata_store/ ``` +Two stores, two pushes, and they must stay **separate roots**. Each package keeps its +own `manifest.json` at its root and does a read-modify-write on it, so merging the two +into one directory would have the producers dropping each other's entries. + `data/` is a third gitignored directory, and most of it does **not** need to travel: - `xls_data` (441M) and `cot_data` (94M) are CFTC archives the ETL downloads from @@ -406,6 +427,11 @@ slated for removal... Refrain from using this package or pin to Setuptools<81."* **Store errors, or every instrument empty.** `COTDATA_STORE` is unset, points somewhere wrong, or the store was never synced. It is read at import. +**`MARKETDATA_STORE is not set`, or price reads fail while COT works.** The second store +is missing. Positioning and prices come from different packages since ADR-0007, so one +can be healthy while the other is absent — and this is the shape that failure takes. +Set it and sync `marketdata_store` the same way as the first. + **Charts render but prices are missing or stale.** Expected if the producer has not run. The server cannot fetch prices; see the top of this file.