From 46dd59f04ce3cac28692ae851c659c04e9f3836a Mon Sep 17 00:00:00 2001 From: "Matt S." Date: Sun, 9 Aug 2026 02:25:52 +0000 Subject: [PATCH 1/2] Declare marketdata, and document the second store the server now needs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit cotmetrics reads bars from marketdata as of its ADR-0007 repoint, so this package needs the dependency declared even though it imports neither: its own source has no cotdata import and no get_prices call at all. Measured rather than assumed — the work order lists cot-analyzer as a consumer to repoint, and there is nothing here to repoint. The coupling is a declared dependency plus a cotdata-update subprocess, and that subprocess stays where it is because COT production did not move. The deployment consequence is the part worth care. The server now needs MARKETDATA_STORE set and a second synced store, and without it the price reads behind the indexer, the rejection scores and the max-pain snapshot fail at the point of read. Positioning and prices come from different packages now, so one store can be healthy while the other is absent; the troubleshooting section names that shape directly, because "COT works but prices do not" is not a symptom this file could previously produce. Both stores are synced rather than produced there — that box cannot produce bars at any Python version, norgatedata needing a Windows-only Norgate Data Updater — and they must stay separate roots, since each package keeps a manifest.json at its own root and does a read-modify-write on it. Also corrects the cache note: the metrics cache busts on both upstream store versions now, and the reason is that the bump it was written for was a price schema bump. 131 tests pass, ruff clean over what CI lints, and the internal dependency floor check passes with all three siblings installed. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Jo4iovRfc2fzE9MwcLp7r2 --- pyproject.toml | 1 + requirements.txt | 6 ++++-- scripts/check_dep_floors.py | 1 + server-side/README.md | 32 +++++++++++++++++++++++++++++--- 4 files changed, 35 insertions(+), 5 deletions(-) diff --git a/pyproject.toml b/pyproject.toml index 0f96d73..8def06a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -33,6 +33,7 @@ dependencies = [ "pyyaml==6.0.3", "requests==2.32.5", "cotdata>=0.1.0", + "marketdata>=0.1.0", # bars; ADR-0007 moved them out of cotdata "cotmetrics[options]>=0.3.0", ] diff --git a/requirements.txt b/requirements.txt index f2c5443..da701a8 100644 --- a/requirements.txt +++ b/requirements.txt @@ -18,10 +18,12 @@ ruff==0.15.22 # Linter — pinned so CI installs it and stays reproduc tomli==2.0.1; python_version < "3.11" # for scripts/check_dep_floors.py on <3.11 # data/metrics layer (editable siblings). cotmetrics provides the COT index, -# signals, indexer + the legacy ETL/options/scheduler; it depends on cotdata. -# Both are published on PyPI (cotdata, cotmetrics), but we install them EDITABLE +# signals, indexer + the legacy ETL/options/scheduler; it depends on cotdata for +# CFTC positioning and, since ADR-0007 moved bars out, on marketdata for prices. +# All three are published on PyPI, but we install them EDITABLE # from sibling checkouts so in-workspace changes are picked up immediately and CI # catches integration breaks before those siblings are released. Pin the PyPI # releases instead if you want reproducible, checkout-free builds. -e ../cotdata +-e ../marketdata -e ../cotmetrics[options] diff --git a/scripts/check_dep_floors.py b/scripts/check_dep_floors.py index f0db7ca..e5956cb 100644 --- a/scripts/check_dep_floors.py +++ b/scripts/check_dep_floors.py @@ -31,6 +31,7 @@ # rather than a package index, so only these need the extra check. INTERNAL = { "cotdata", + "marketdata", "cotmetrics", "crucible", "crucible-stack", diff --git a/server-side/README.md b/server-side/README.md index af4f902..cdc434e 100644 --- a/server-side/README.md +++ b/server-side/README.md @@ -140,12 +140,23 @@ chmod 600 /root/trading_workspace/cot-analyzer/.env **Required:** ```bash -COTDATA_STORE=/root/cotdata_store # the synced price store, see step 6 +COTDATA_STORE=/root/cotdata_store # synced CFTC positioning store, see step 6 +MARKETDATA_STORE=/root/marketdata_store # synced daily-bar store, see step 6 ``` Nothing works without `COTDATA_STORE`. Every entry point resolves instruments through it, so a missing or empty store fails at import rather than degrading. +**`MARKETDATA_STORE` is required too, and it is new.** ADR-0007 makes `cotdata` CFTC +positioning only and moves every bar to `marketdata`, so the price reads behind the +indexer, the signal rejection scores and the options max-pain snapshot now resolve +against a second store. It fails the same way the first one does — by name, at the +point of read — rather than serving an empty frame that reads as "no data for this +instrument". + +Both are **synced**, not produced here. This box cannot produce bars at all: +`norgatedata` drives a locally installed Norgate Data Updater and NDU is Windows-only. + **Required for the emailed Signal Matrix report:** ```bash @@ -215,8 +226,11 @@ They are cotmetrics runtime state, and PR #12 moved them out of this repo into `.local-state/cot-analyzer/`, so the deprecated script could not have shipped them anyway. **The server rebuilds the cache itself** — confirmed by the maintainer on 2026-08-04, not inferred from the push having been broken. `CotIndexer` writes per-instrument parquet under -`COTMETRICS_CACHE` on first use and busts it on the upstream `cotdata.schema_version()` plus -`METRICS_CACHE_VERSION`, so a store push is the only input it needs. +`COTMETRICS_CACHE` on first use and busts it on BOTH upstream store versions +(`cotdata.schema_version()` and `marketdata.schema_version()`) plus +`METRICS_CACHE_VERSION`, so a store push is the only input it needs. Both are watched +because the case that guard was written for — reconstructed volume being promoted — was +a *price* schema bump, and prices now live in the second store. The first sync after a release is therefore slower than steady state, because the cache is cold. That is a latency cost on one request, not a missing payload. @@ -226,8 +240,15 @@ The store push by hand, for reference, is: ```bash rsync -avz --no-o --no-g --progress --exclude='.DS_Store' \ /path/to/cotdata_store/ USER@HOST:/root/cotdata_store/ + +rsync -avz --no-o --no-g --progress --exclude='.DS_Store' \ + /path/to/marketdata_store/ USER@HOST:/root/marketdata_store/ ``` +Two stores, two pushes, and they must stay **separate roots**. Each package keeps its +own `manifest.json` at its root and does a read-modify-write on it, so merging the two +into one directory would have the producers dropping each other's entries. + `data/` is a third gitignored directory, and most of it does **not** need to travel: - `xls_data` (441M) and `cot_data` (94M) are CFTC archives the ETL downloads from @@ -406,6 +427,11 @@ slated for removal... Refrain from using this package or pin to Setuptools<81."* **Store errors, or every instrument empty.** `COTDATA_STORE` is unset, points somewhere wrong, or the store was never synced. It is read at import. +**`MARKETDATA_STORE is not set`, or price reads fail while COT works.** The second store +is missing. Positioning and prices come from different packages since ADR-0007, so one +can be healthy while the other is absent — and this is the shape that failure takes. +Set it and sync `marketdata_store` the same way as the first. + **Charts render but prices are missing or stale.** Expected if the producer has not run. The server cannot fetch prices; see the top of this file. From 324f0a1d0f55b8024c754922297e7f364e2c5d3b Mon Sep 17 00:00:00 2001 From: "Matt S." Date: Sun, 9 Aug 2026 02:28:43 +0000 Subject: [PATCH 2/2] CI: check out marketdata, since it is not on PyPI MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every job went red at install. This repo resolves its internal siblings by checking them out and installing editable, and marketdata has no PyPI release, so declaring the dependency without adding the checkout makes `pip install -r requirements.txt` fail before a single test runs. A dummy MARKETDATA_STORE joins the dummy COTDATA_STORE for the same reason the first one exists: neither package defaults a missing store root to somewhere plausible, both raise by name, and there are two roots now. Worth noting the shape rather than just the fix. The local run passed because the sibling was already installed in the venv; CI builds the environment from scratch, which is the only place a missing distribution shows up. That is the same class as the three defects this port has already produced — a green run proving something narrower than it appeared to. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Jo4iovRfc2fzE9MwcLp7r2 --- .github/workflows/python-test.yml | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/.github/workflows/python-test.yml b/.github/workflows/python-test.yml index 1438abc..919b751 100644 --- a/.github/workflows/python-test.yml +++ b/.github/workflows/python-test.yml @@ -30,6 +30,15 @@ jobs: repository: mspinola/cotdata path: cotdata + - name: Check out marketdata (daily bars — resolves `-e ../marketdata`) + # ADR-0007 moved bars out of cotdata, and marketdata is not on PyPI, so this + # sibling checkout is the only thing that resolves the dependency cotmetrics + # now imports. + uses: actions/checkout@v4 + with: + repository: mspinola/marketdata + path: marketdata + - name: Check out cotmetrics (metrics/data layer — resolves `-e ../cotmetrics`) uses: actions/checkout@v4 with: @@ -48,7 +57,7 @@ jobs: python -m pip install -r requirements.txt - name: Verify internal dependency floors - # cotdata / cotmetrics are installed editable from ../ at whatever HEAD was + # cotdata / marketdata / cotmetrics are installed editable from ../ at whatever HEAD was # checked out, so pip never enforces the version floors in pyproject.toml. # This asserts the installed siblings actually satisfy those declared floors. working-directory: cot-analyzer @@ -62,9 +71,12 @@ jobs: working-directory: cot-analyzer env: PYTHONPATH: src - # Dummy store so cotdata's COTDATA_STORE guard doesn't error at import. + # Dummy stores so each package's store guard doesn't error at import. + # Two of them since ADR-0007: positioning and bars are separate roots, and + # neither package defaults a missing root to somewhere plausible. COTDATA_STORE: /tmp/cotdata_store + MARKETDATA_STORE: /tmp/marketdata_store COTMETRICS_CACHE: /tmp/cotmetrics_cache run: | - mkdir -p /tmp/cotdata_store /tmp/cotmetrics_cache + mkdir -p /tmp/cotdata_store /tmp/marketdata_store /tmp/cotmetrics_cache pytest tests/